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ABSTRACT 

Training Opportunities in the Private Sector (TOPS) 
was a small-scale, voluntary program operated as one among many 
employment-related options available to recipients of Aid to Families 
with Dependent Children in Maine's Work Incentive (WIN) Program. TOPS 
was distinguished from Maine's other WIN demonstration activities by 
three program elements. It was a prescribed sequence of 
activities — prevocational training, unpaid work experience, and 
subsidized on-the-job training, preferably in the private sector. 
Second, it was intended to reach hardei — to-employ clients. Third, it 
tended to involve more intensive use of staff time. A comparison of 
the experiences of an experimental group of TOPS participants with a 
control group that did not have access to the TOPS sequence indicated 
that TOPS produced employment and earnings gains that contained 
throughout the 3-year follow-up period. The combination of earnings 
gains and no welfare reductions led to substantial increases in the 
total measured income of TOPS enrollees. Because of the relatively 
high cost of the program and the absence of welfare savings, however, 
TOPS did not pay for itself from the perspective of government 
budgets within a 5-year period. (MN) 
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PREFACE 



This is the final report on MDRC's evaluation of Maine's Training 
Opportunities in the Private Sector (TOPS) Program, TOPS offered a 
prescribed sequence of activities that included pre- vocational training and 
unpaid work experience and ended in on-the-job training (CJT) positions. 
Operated as a snail-scale, voluntary program, it was one of many 
employment-related options available to recipients of Aid to Families with 
Dependent Children (AFDC) in Maine. 

The TOPS evaluation provides the first opportunity for MDRC to examine 
the results of a relatively intensive voluntary program in its multi-state 
Demonstration of State Work/Welfare Initiatives. The other programs 
evaluated to date in MDRC's multi-state project (in Arkansas, California, 
Illinois, Maryland, Virginia, and West Virginia) were all mandatory, 
generally lew- to-moderate- cost initiatives that served a broader segment of 
the WIN caseload. TOPS is also of interest because, through an innovative 
funding process known as grant diversion, it utilizes AFDC funds to cover 
part of the cost of the CJT positions. The final report in the MDRC 
Work/Welfare Demonstration, to be released this fall, will study another 
voluntary, grant diversion-funded OJT program in New Jersey. 

The Demonstration of State Work/Welfare Initiatives is a unique 
opportunity for MDRC to work with states in evaluating their employment 
programs and to examine a subject of national and state concern: the 
critical relationship between work and welfare dependency. Addressing 
state issues in a manner that benefits policy at many levels is a challenge 
that MDRC is privileged to undertake. 



This demonstration documents a shift in responsibility fron the 
federal government to the states. The individual studies evaluate the 
initiatives designed and implemented by the states themselves under the 
provisions of the Onnibus Budget Reconciliation Act of 1981 (CORA), which, 
for the first time, authorised states to operate Community Work Experience 
Programs (CWEP) for AFDC and to streamline the administration of their Work 
incentive (WIN) systems, since states responded to these options in 
different ways, the demonstration is not built around a single model. 
Rather, the evaluations focus on initiatives that represent some of the 
major variations being tried in this country, spanning a range of local 
economic conditions and AFDC provisions. 

MDRC could not have conducted this multi-year study without the 
support of The Ford Foundation, which provided funds for planning and for 
the evaluation activities of the participating states, matching an equal 
investment of state or other resources. This joint funding is another 
significant aspect of the demonstration. 

Throughout this demonstration MDRC has been gratified by the 
commitment of the participating states and foundations and their interest 
in the findings, it is our hope that this demonstration and its results 
have contributed to informed decision-making and will ultimately lead to 
even more effective programs, which will increase the self-sufficiency of 
welfare recipients. 



Judith M. Gueron 
President 
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EXECUTIVE SUMMARY 



This is the final report on the State of Maine's Training Opportun- 
ities in the Private Sector (TOPS) Program, TOPS was a small-scale, 
voluntary program operated as one among many employment-related options 
available to recipients of Aid to Families with Dependent Children (AFDC) 
in Maine's Work Incentive (WIN) Demonstration. Thus, the evaluation 
focuses on one program alternative, not the WIN Demonstration system as a 
whole. 

TOPS was distinguished from Maine's other WIN Demonstration activities 
by three program elements. First, it was a prescribed sequence of activi- 
ties — pre-vocational training, followed by unpaid work experience, 
follwed by subsidized on-the-job training (OJT), preferably in the private 
sector. Second, it was intended to reach a harder- to- employ group among 
the AFDC caseload, as measured by prior welfare dependency and relative 
lack of work experience, although candidates had to apply for the program 
and go through additional screening before being judged appropriate to 
participate. Third, TOPS tended to involve more intensive use of staff 
time* 

In addition to these programmatic differences, TOPS differed from 
other WIN Demonstration activities in Maine in that it used grant diversion 
to finance part of the wage subsidies offered to employers who hired 
participants in the OJT component. Grant diversion is the financial 
mechanism through which all or part of a public assistance grant is used to 
finance program services for recipients. 



The evaluation compared the experiences of an experimental group which 
was enrolled in TOPS to those c, a control group which ciid not have access 
to the TOPS sequence. It found that TOPS produced employment and earnings 
gains that continued throughout the three year follow-up period. Most of 
the earnings gains reflected some combination of higher wages or more hours 
worked, rather than merely increases in the percentage of individuals 
employed. TOPS did not have an impact on average welfare receipt, however. 
The combination of earnings gains and no welfare reductions led to 
substantial increases in the total measured income of TOPS enrollees. But, 
due largely to the relatively high cost of the program and the absence of 
welfare savings, TOPS did not pay for itself from the perspective of 
government budgets within a five year period. 

The TOPS Program Design 

The Maine WIN Demonstration is called the Welfare Employment, Educa- 
tion and Training Program (WEET). The state's regular WIN program was 
replaced by WEET in April 1982, following passage of the Federal Omnibus 
Budget Reconciliation- Act (OBRA) in August 1981. OBRA gave states 
increased flexibility in designing WIN — the major federally-funded 
employment and training program for the AFDC population — and authorized 
the use of welfare grant diversion. 

TOPS was operated as an experimental program within WEET, beginning in 
October 1983. Program intake ended in December 1984, but TOPS continued 
into 1986, as enrollees finished their OJT positions. Since December 1984 
the TOPS sequence has not been offered as a regular part of WEET, although 
similar components are available to WEET registrants on an individual 
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basis. 



TOPS was targeted to single heads-of-household who had been on welfare 
for at least six months, were not employed at enrollment, an£ applied to 
participate, Ths women were subject to further screening by WEET staff w!»o 
exercised considerable discretion in deciding whether a potential client 
was appropriate" for TOPS. Screening included identification of any 
barriers to participation (such as child care, health or basic literacy 
problems). In addition, sane offices responding to concerns about the 
level of QJT placements tried to increase the placement rate by applying 
more rigorous screening criteria regarding motivation and educational 
attainment. 

The three phases of the TOPS sequence worked as follows: 

!• Pre-vocational training lasted two to five weeks and stressed 
personal growth as well as job-seeking and job-holding skills. 

II * Work experience consisted of 20-hour per week, unpaid posi- 
tions in the public or non-profit sector for up to 12 weeks. 
The intent was to teach good work habits and provide partic- 
ipants with an employer reference. It was offered to 
participants who completed Phase I but were judged not to be 
ready for an OJT placement. 

Ill* On-the-lob training consisted of placement into subsidized 
training positions, primarily in the private sector. The 
training period was a maximum of six months, and the employer 
subsidy was set at 50 percent of the new onployee's wages. 
Participants who demonstrated their motivation and acquisition 
of basic work skills became candidates for OJT. 

Participants in Phases I and II continued to receive their AFDC 
grants, were paid training-related allowances, and were eligible for basic 
education on a limited batis. Enrollees were not supposed to engage in any 
other education activities. 

As noted, a partial funding mechanism for the OJT component in Phase 




Ill was grant diversion. Welfare recipients were placed in regular wage- 
paying jobs; the amount of AFDC they received, if any, was calculated just 
as it would have been for any employed AFDC recipient. The difference 
between the amount of the AFDC grant a recipient had been receiving before 
going to work and the amount paid to her after she started working was 
diverted into a wage pool and used to subsidize the CJT positions. WEET 
staff wanted to experiment with grant diversion because putting AFDC funds 
into an employment-related activity was regarded as a premising investment, 
and because it represented a new source of funding in an era of shrinking 
federal resources. 

TOPS was offered in a series of discrete cycles, each consisting of 
the three components. The start of the pre-vocationai training was 
considered the start of a cycle, and the women who began pre-vocational 
classes together were treated as a distinct group by staff, even though 
they moved through the later components in the sequence at varied paces, 
in all, 30 TOPS cycles were offered at 15 different locations throughout 
the state. 

The intent behind TOPS' carefully sequenced set cf activities — 
designed to address problems arising from AFDC recipients' low self-esteem 
as well as their lack of work experience - was to help the women obtain 
jobs that paid more than the minimum wage and offered opportunities for 
advancement. Ultimately, these jobs were expected to enable enrollees to 
move off welfare entirely. This was a particular concern in Maine where, 
historically, a relatively large proportion of the caseload worked full 
time and still received AFDC benefits. 

Central WEET staff recognized that their system did not have the 
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financial resources or the staff capacity to operate the full TOPS 
sequence. Therefore, they contracted with the state's major training 
system for the disadvantaged — authorized under the Job Training 
Partnership Act (JTPA) — to provide most of the Phase I and some of the 
Phase in services and to help underwrite the CJT subsidy. 

The Research Design 

MDRC 1 s evaluation of TOPS examined the experiences of all female 
heads-of-household accepted for TOPS throughout the period of its 
operation. In order to evaluate the impact of TOPS f individuals were 
randomly assigned to an experimental or a control group. Those in the 
experimental group were eligible for the full TOPS sequence? the control 
group was excluded frcm the specific TOPS sequence but remained eligible 
for all other WEET services, which included sane components similar to 
those in TOPS. 

There is a widespread consensus that random assignment is the most 
reliable way to measure the effects of a welfare employment program. This 
is because it ensures that the group receiving the treatment being evalu- 
ated is similar to the control group except for the program activities to 
which they are assigned. Outcomes for the control group serve as a 
benchmark against which outcomes for the treatment group can be measured. 
Statistically significant differences in outcomes for the groups during the 
follow-up period can be confidently attributed to the effects of the 
treatment itself because the experiences of the control group reflect the 
extent to which program registrants would have become employed or left 
welfare in the absence of the treatment. In this study, differences are 



considered statistically significant if there is no more than a 10 percent 
possibility that they could have occurred bv chance. 

Two other aspects of the research design should be considered in 
interpreting the TOPS findings. First, the impact results - i.e., the 
differences in outcomes for experimentals and controls — reflect in part 
the substantial amount of activity by controls in jobs and in employment- 
related programs. This level of accivity was caused by several factors: 
the research sample consisted exclusively of people who volunteered for 
TOPS and were not screened out; controls were eligible for all WEET 
services; the economy in :*iine was improving during the demonstration 
period; and the work incentives embodied in Maine's AFDC benefit structure 
are greater than in most states. 

Second, even though all individuals accepted for TOPS were included in 
the research sample, the program was small and, therefore, the research 
sample was also small — 444 women. Small sample sizes reduce the chance 
that outcome differences of a given size will be statistically significant. 

Evaluation Sample and Data 

Of the 444 sample members, 2 97 clients were randomly assigned to the 
experimental group and 147 to the control group. Two experimentals were 
assigned for every control because WEET program operators felt that it was 
important to serve a large proportion of those recruited. The small sample 
size reflected problems in stimulating interest in the specific TOPS 
sequence as opposed to alternative activities, combined with staff selectiv- 
ity in enrolling clients. it does not reflect lack of interest in * 
receiving services on the part of AEDC recipients in Maine. 
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Nearly all TOPS sample members were white and over half were divorced 
or widowed. Half had children under the age of six. In keeping with TOPS 
program goals, most of the sample members were relatively long-term AFDC 
recipients — 63 percent had been receiving AFDC for more than two years — 
and only one-third had recent employment experience. However, due to the 
additional screening criteria developed by local staff, sample members were 
relatively well educated: 75 percent had received a high school diplaoa or 
GED by the point of random assignment. 

in addition to their high level of education, the fact that 61 percent 
of the TOPS sample members were not required to register for employment 
services with WEET (mostly because they had young children) but did so 
anyway suggests that they were a highly motivated group with few barriers 
to participation. Thus, although these women would be characterized as 
harder-to-employ in terms of their welfare histories and limited recent 
employment experience, there are indications that the program concentrated 
on a segment within this group with somewhat more favorable enployment 
prospects. In addition, a number of factors — the high percentage of 
sample menbers who were divorced or widowed, the presence of pre-school 
children, the relatively high level of educational attainment and the 
absence of employment history -- suggest that the wanen in the TOPS sample 
may represent a displaced hanemaker population, that is, wcmen who discon- 
tinued or postponed working for marriage and family and then experienced 
hard times. 

Evaluation data on the research sample came from several sources. 
Client characteristics data came frcro Client Information Sheets filled out 
by WEET staff at randan assignment, Data on TOPS program participation 
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came from the WEET information system which, for non-TOPS activities, was 
supplemented by the JTPA information system. Follow-up data on earnings 
and AFDC payments came ■ » the state's Unemployment insurance system and 
AFDC payments system, respectively. Qualitative information, ' mainly 
interviews with program staff, was used to augment the quantitative data. 

Findings On Partici pation and Program Implementation 

• Program scale in TOPS remained small and fell below the level 
anticipated. 

A total of 2 97 experimentals were enrolled in TOPS over the 15-raonth 
intake period, well below the enrollment target. This compares with a WEET 
caseload of 3,157 registrants shortly before the start of the TOPS demon- 
stration and a Maine AFDC caseload of 16,556 heads-of-household in January 
1983. Both the snail absolute scale of the program and the failure to 
reach the anticipated level were typical of other grant diversion OJT 
programs studied by MDRC. This suggests that OJT programs iunded with 
grant diversion are likely to remain only a limited part of a broad-based 
welfare reform strategy. 

• Approximately 90 percent of the TOPS experimentals partici- 
pated in at least one of the three TOPS components. Partici- 
pation was highest in the pre-vocationai training and lowest 
in OJT; the most frequent combination of services was 
pre-vocational training followed by work experience. 

Due at least in part to the voluntary nature of TOPS and the screening 

which excluded applicants with potential barriers to participation, a high 

percentage of experimentals participated in employment-related activities. 

As would be expected in a sequential program, participation was highest in 

the first activity in the sequence, and declined in each subsequent compon- 
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ent, as enrollees encountered practical problems, found employment, went 
off AFDC, lost interest in the program, decided to pursue other training 
activities, or were found inappropriate for subsequent components. Even 
so, nearly 7 0 percent of the experimentals participated in at least two 
TOPS components and about 30 percent participated in OJT in combination 
with another activity. In addition, 32 percent were active in job search, 
although this was not part of the program design, and a small proportion of 
experimentals participated in other education and training activities. 
Only two experimentals were sanctioned for noncompliance. 

• Just over half of the control group also participated in 
employment activities, which typically were less intensive 
than the combination of TOPS services received by the 
experimental group. 

Since members of the control group were eligible for other services 

x 

and were to be treated like other WEET clients, WEET staff worked actively 
with them. This included identifying and arranging appropriate services, 
referring them to education or training activities available through other 
vendors, and providing support services. About a quarter of the controls 
participated in job search, and a quarter were active in education or 
training. Although a substantial proportion — almost a quarter — did 
participate in TOPS-like pre-vocational training, work experience " or OJT, 
none got the sequential approach or all three of the activities that 
defined TOPS. Nor were the activities engaged in by controls as intensive, 
on average, as those in the TOPS sequence. 

• She total estimated cost of the TOPS agram came to $2,627 
per experimental. However, because controls received some 
services, the net cost of all employment and training services 
was $2,244 per experimental. 

This was the highest estimated net cost among the welfare employment 
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programs MDRC has recently evaluated. The cost of TOPS itself — $2,627 
per experimental - represents $2,933 per person who participated in the 
program. The cost includes program operating expenditures ($1,613 per 
experimental), as well as allowances, support services and wage subsidies 
($1,014 per experimental). m addition, experimental received non-TOPS 
services from WEET, JTPA and other providers that cost an estimated $134 
per experimental. The control group, cn the other hand, received $517 
worth of services, on average. 

• The TOPS sequence depended on interagency coordination between 

m^*'^^ and JWA respite improvements over 

time, differences in philosophy and probleas in communication 
complicated program implementation. 

in Maine, WEET and JTPA staff had quite different philosophies about 

whether and how to work with AFDC recipients. These differences were 

exacerbated by initially poor communication channels between tb» two 

systems. Maine's experience suggests that several items are important to 

interagency coordination in a multi-component sequence of activities: 

involvement at the outset by local as well as central staff from both 

agencies; designation of a liaison in each local office; joint staff 

training,- regular opportunities for case-management reviews with staff from 

both agencies; and continued attention from top staff in both agencies. 

implementation of TOPS was also complicated by the disjuncture between the 

prescriptiveness of the TOPS sequence and the greater flexibility afforded 

staff within regular WEET to tailor services to individual circumstances. 

* 2S ° £ th * j ° b dtvelo f*«t for OJT positions was done by the 
TOPS participants themselves, applying lessons learned in the 
pre-vocational phase of their training. 

Both JTPA and WEET staff came to rely heavily on client Job search as 
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a source of OJT positions, although this had not been envisioned in plan- 
ning the demonstration. During some of the TOPS cycles, participants were 
Placed in regular job clubs; in others, they engaged in independent job 
search under the supervision of job development staff. (Some enrollees got 
unsubsidized jobs through this method.) Only the most promising OJT candi- 
dates were referred directly to employers in the staff's pool of available 
positions; on the whole, there was little individualized job development. 
Rollover from the work experience positions was another source of CJT 
positions. 

• The OJT positions att the program goal of paying a starting 
wage of $4.00 per hour, on average. Tni Majority of the OJT 
positions relied over into UMObsidized employment with the 
same employer. 

On average, toe OJT positions paid $4.04 per hour to start; one- third 
paid more than $4.00, and two-thirds paid $4.00 or less, r'er 70 percent 
were in clerical or service positions — occupations defined as 'typically 
female; and with correspondingly low wage levels. Despite the emphasis on 
private sector placements, just over one-quarter of the OJT contracts were 
written with non-profit employers or public agencies. A review of the job 
descriptions in the OJT contracts suggests that most offered only limited 
training potential. 

The average scheduled length of the OJT positions was 19 weeks; 37 
percent were scheduled for the maximum length of six months. State- 
reported data show that 69 out of 89 OJT participants completed the number 
of training hours called for in their contract; 63 continued in unsubsi- 
dized employment with the same employer. Situational problems and 
dissatisfaction with the position were common reasons for failure to 
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complete an OJT; only a few clients were fired by the employers. 

• In general, the grant diversion process operated well. 
However, funds diverted from AFDC grants covered only about 
half of the OJT wage subsidy, less than the 70 percent 
anticipated. 

Despite sane problems in synchronizing the timing of the diversion of 
AFDC funds into the wage pool with changes in OJT participants' AFDC 
grants, the grant diversion process generally worked smoothly in TOPS. The 
experience suggests that careful monitoring is critical; that it would be 
difficult to operate a grant diversion program on a large scale if a 
combination of automated and manual record-keeping is required; and that 
the actual amount of funds available through grant diversion is difficult 
to predict in advance. it may therefore be necessary to have additional 
sources of revenue available to cover any short-falls in the wage pool. 

• The TOPS experience suggests that the OJT subsidy «ay have 
been «ore of a hiring bonus for employers than compensation 
for higher training costs or lower productivity. 

A combination of factors — especially that the OJT positions seemed 
to offer little training beyond that given to any new employee, and that 
employers found the OJT hires to be about as productive as other new hires 
— suggests that the OJT subsidy was not necessary to compensate for any 
added training costs or reduced productivity, it does not follow, however, 
.that an OJT is inappropriate even if employers benefit financially. The 
added cost of the subsidy might be worthwhile if it accelerates the speed 
with which AFDC recipients are placed in lobs, or leads to better jobs than 
would otherwise have been available to them. 
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frinf Un gc oa Program impacts 

The impact findings described below should be interpreted in light of 

the following key points. First, the discussion occasionally refers to the 

; in- program; and ;post-prograia; periods. The former refers to the first 

six quarters of follow-up after random assignment, when a substantial 

percentage of experimentals vere in a TOPS component; the latter period 

begins with the seventh quarter of follow-up, when only six percent of 

experimentals were still in OJT positions. Second, the findings reflect 

the impact of the mix of services experimentals received, and do not 

.measure the separate effect of (XFT. Third, the earnings for experimentals 

include wages paid in subsidized OJT positions? however, the subsidized 

wages did not have any materia. 1 , effect on experimentals* earnings after 

quarter 6. And finally, the APDC expenditures for experimentals reflect 

only the payments made directly to individual recipients; tha amounts 

diverted into the wage pool are, however, taken into account as part of the 

benefit-cost analysis discussti «acer. 

• TOPS achievtd sustained earnings increases over the 11-quarter 
follov-up period. 

Over the full follow-up period (quarters 2 through 11), TOPS increased 
earnings on average by $1,745 per experimental, over and above the control 
group average of $5,599. (See Table 1 and top panel of Figure 1.) During 
the in-program period (through q*-rter 6), earnings of experimentals 
averaged $2,855 and earnings of controls averaged $2,300, yielding a posi- 
tive but not statistically significant difference of $555. During the 
post-program period (quarters 7 through 11), TOPS increased average 
earnings by a statistically significant $1,190 per experimental, over and 



above the control group average of $3,298. 

• Experimental* were employed for aore follow-up quarters than 
controls, and «ore experimentals than controls were employed 
In each quarter except the first two. Moat of these differ- 
ences were not statistically significant, probably because of 
small sample sizes. 

Beginning with the third quarter, experiment als had consistently 
higher rates of employment than controls, with the differences ranging fron 
1.1 to li.i percentage points. (See Table 1.) For the full follow-up 
period, experimentals were employed on average for 4.69 quarters versus 
4.21 quarters for controls, yielding a not statistically significant 
difference of almost half of a quarter. 

• Three-quartara of the overall earnings impact in the post- 
program period was due to increased wage rates or hours worked 
for those who were employed, rather than to a higher 
proportion of experiaentals who were ever employed. 

Over the full follow-up period, only 7 percent of the earnings impact 

was due to increased rates of employment, and 19 percent to an increase in 

the number of quarters with employment for these who were ever employed. 

Nearly 75 percent of the earnings impact, therefore, was due to increased 

earnings of experimentals in the quarters in which they were employed. 

This reflected some combination of higher wages per hour, more hours per 

week, or more weeks worked per quarter. However, the data do not permit an 

•estimation of the relative importance of these three factors. 

• TOPS did not reduce welfare receipt over the full f,oliow-up 
psriod, the in-program program period or the post-program 
period. 

In most quarters, slightly more experimentals than controls received 
welfare, although the differences were never statistically significant. 
(See Table 2 and bottom panel of Figure 1.) over the full follow-up 



TABLE 1 
MAINE 

IMPACTS OF THE TOPS PROGRAM ON EMPLOYMENT AN 0 EARNINGS 



Outcome and FoLLow-Up Period 


Experimental 8 


Control s 


01 f f er once 


Ever Employed 








Quarters 2-11 8 


61 .8% 


80. M 


1 .6X 


Quarters 2-6* 


71,2 


85.6 


5.6 


Quarters 7-11 


66.7 


65.3 


3.4 


Ever Employed 








Qua r te r 


of Random Assignment 


16.4 


24.7 


-8.2" 


Quer ter 


2 


23 .5 


34.8 


-11 • 3 


Q ue r te r 


3 


42.8 


39.3 


3.5 


Quarter 


4 


52.1 


43 .8 


8.3 


Quarter 


5 


49.9 


41 .3 


8.6* 


Quarter 


6 


47.9 


40.7 


7.2 


Quer ter 


7 




Ho • U 


O . *T 


Qua r te r 


B 


50.2 


43.4 


6.9 


Quer ter 


9 


50 .6 


43.4 


7.1 


Quer te r 


10 


50.7 


39.7 


11.1" 


Quarter 


11 


50.4 


49.3 


1.1 


Average Total Eernlngs 








Quj r te r £ 


2-M° 


$7344.00 


$5599.17 


$1744. 83" 


Quer te r g 


2-6* 


2655.39 


2300.89 


554.50 


Quer te r e 


7-11 


4466.61 


3296.28 


1 1 90.33" 


Average Total Earnings 








Quar te r 


of Rendom Assignment 


70.97 


108.74 


-37.78 


Quer te r 


2 


I / c . 0 0 


QQQ CO 


_ A A C Q7I| 

— 1 1 O • O / * * 


Q ue r te r 


3 


535.12 


436 .18 


98.94 


Quer te r 


4 


666.94 


528.54 


1 58. 40* 


Q uer te r 


5 


715.91 


475.37 


240. 54" 


Q uer te r 


6 


744.78 


572.28 


17 2.50* 


Q uer te r 


7 


639.99 


601 .23 


238. 76" 


Q ue r te r 


B 


861 .97 


662.55 


219.42" 


Quarter 


9 


917.52 


666.60 


248.92" 


Q ue r te r 


10 


915.73 


641 .63 


274. 10" 


Quer tor 


11 


933.40 


724.28 


209.13* 
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NOTES: These calculations Include zero valuee for seaple nenbere not 
employed. There aey be eoae discrepancies 1n suns end differences due to 
rounding. 

U two-te1led t-test ves applTed to er difference between 
expariraentel end control groups. Statistical sign. ;ance levels ere Indicated 
as follows: * = 10 percent; = 5 percent; "• = 1 percent. 

e 

Querter 1# the quarter of rendom essl gnmenti may conteln sorae 
Income froo the period prior to rendom esstgnaent end 1s therefore excluded from 
cum ul e t1 v e outcomes. 
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TABLE 2 

MAINE 

IMPACTS OF THE TOPS PROGRAM ON AFOC RECElK 



Outcome and Follow-Up Period 



Experimented Controls Difference 



Ever Received AFDC 
Querters 2-1 1 8 
Quarters 2-6 6 
Quarters 7-11 



Ever Received AFOC 

Quarter of Random Assignment 

Quarter 2 

Quarter 3 

Quarter 4 

Quarter 5 

Quarter 6 

Querter 7 

Querter B 

Querter 9 

Que r t e r 10 

Querter 11 



Averege Totel AFOC Peymetits 
Querters 2-11° 
Querters 2-6° 
Querters 7-11 



Averass AFDC Peyments 

Querter of Random Assignment 
Querter 2 
3 
4 
5 
6 
7 



Que r te r 
Que r te r 
Que r te r 
Que r te r 
Que r te r 
Querter B 
Querter 9 
Querter 1 0 
Qua r te r 1 1 



9B. 3% 
98. 3 
73.3 



98.6 
97.6 
89.9 
81 .8 
77.3 
73.8 
70.0 
64.4 
62.8 
53.9 
55.8 



$6768.45 
3870.62 
2897.84 



924.25 

91 8.20 

834.50 

741.90 

694.64 

681 .38 

636.64 

595.98 

578.21 

561 .67 

525.34 



98. OX 

98.0 

71.6 



99.4 
98.0 
89.8 
85.1 
77.1 
71.4 
68.7 
63.1 
60.2 
52.5 
49.8 



$6598.52 
3750.36 
tJ848.1 6 



941 .22 
886.79 
798.73 
728.16 
686.92 
S49.76 
636 .23 
607.13 
582.83 
516.42 
505.55 



0.3% 

0.3 

1.8 



-0.7 
-0.4 
0.0 
-3.3 
0.2 
2.4 
1 .3 
1.3 
2.5 
6.4 
6.0 



$169.93 
120.26 
49.67 



-16.97 
31 .41 
35.77 
13.74 
7.72 
31 .62 
0.41 

-11.15 
-4.62 
45.25 
19.79 



NOTES: These celculatlons include zero values for sample members not 
receiving AFDC. There me y be some di scr apenc 1 e s in sums end differences due 
to round i ng . 
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A tiro-tailed t-test wes applied to eech difference between 
experimentel end control groups. Statistical significance levels ore 
indiceted es follows: • =10 percent; »» = 5 percent 1 percent. 

eQuerter 1 ' the garter °f rendom assignment, lfl ay contein some 
income from the period prior to rendom assignment end is therefore excluded 
from cumuletive outcomes. 
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FIGURE 1 
MANE 

TRENDS IN EARNINGS AND AFDC RECEIPT. BY RF3EARCH GROUP 





Quarter Relative to Random Assignment 
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Period, 98.3 percent of experimental* received welfare at sane point, 
versus 98.0 percent of controls. The number of months on welfare averaged 
19.6 for experimentals, versus 19.2 for controls. Total AFDC income over 
the period averaged $6,768 for experimentals, versus $6,598 for controls, 
yielding a not statistically significant increase of $170. 

There is no obvious explanation for the lack of impact on welfare 
receipt, given the consistent earnings impacts. Scrutiny of the AFDC data 
indicated that systematic errors in grant calculations were not the answer. 
Three factors seem plausible. First, Maine's AFDC program allows earnings 
up to 27.5 percent of the standard of need to be excluded from the grant 
calculation as a work incentive feature. Any TOPS impacts in this earnings 
range would not have reduced welfare. Second, any impact for experimentals 
above the point where earnings were already sufficient to remove them from 
the welfare rolls could not generate any further AFDC savings. Third, the 
pre-vocational component of TOPS included discussions of how going to work 
would affect participants' AFDC grant, a d how they could best be their own 
advocates within the system. This could have led experimentals to be more 
effective than controls in making sure their grants were not affected 
inappropriately. 

• TOPS led to statietically significant increases in total 
■easured incoae orer the full follow-up period, the in-prograa 
period and the post-prograa period. 

Over the full follow-up period, TOPS increased measured income (i.e., 

earnings plus AFDC payments) per experimental by $1,915, on a control group 

average of $12,198. During the in-program period the increase was $675, on 

a control group average of $6,051. During the post-program period, the 

increase was $1,240, on a control group average of $6,146. Total measured 
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incane per experimental increased significantly in every quarter except the 
first two (when controls temporarily had higher earnings than experimen- 
tals, many of whom were in the pre-OJT components of the TOPS sequence). 
As earnings increased and AFDC income held roughly constant, earnings grew 
as a proportion of total income. 

Findings froa the Benefit-Cost Analysis 

The benefit-cost findings measure the overall gains and losses gener- 
ated by TOPS. It is important to remember that these estimates, like the 
impact estimates, present the net benefit-cost picture for TOPS compared to 
the benefits and costs of the substantial employment-related activities 
engaged in by controls. 

This analysis extends the impact results in several important ways. 
First, it includes not only the program's impacts on earnings and welfare 
payments, but also the effects on fringe benefits, tax payments, Medicaid, 
Food Stamps and the administrative costs associated with these transfer 
programs. These effects, which could not be measured directly, are imputed 
from observed earnings, welfare effects and other information. 

Second, using a number of assumptions, the analysis projects program 
benefits and costs that are likely to occur after the end of data 
collection. This longer-range view is necessary because most costs are 
incurred early, when participants are still active in the program, whereas 
benefits can be expected to accrue over a longer time period as individuals 
continue to work and pay taxes. The benefit-cost estimates, thus, extend 
over a five-year period for each sample member, starting with the date of 
random assignment, and include both observed and projected effects. 
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Third, the benefit-cost analysis is concerned with how gains and 
losses differ depending on the perspective of different groups in society. 
Several questions are addressed. For example, do members of the 
experimental group become financially better off as a result of TOPS; and 
do government budgets show net gains or losses due to the program? 

It is important to recognize that while this analysis is comprehen- 
sive, it cannot take inn account all factors that are potentially 
important in interpreting benefit-cost results. For example, it does not 
include the possible displacement of other workers by any increased 
employment of experimental s, or the intangible benefits associated with 
society's preference for work over welfare. 

• Over a five-year period, TOPS enrollees benefited by an 
average of more than $3,000 per experimental. 

As shown in the top panel of Table 3, the estimated net present value 
of TOPS to the welfare recipients who enrolled in it was $3,182 per experi- 
mental. This reflects steady gains in earnings and fringe benefits — from 
both OJT and regular jobs ~ which were offset somewhat by taxes, but not 
by reductions in transfer payments. Indeed, estimated AFDC and Medicaid 
payments actually increased, although Food Stamps and Unemployment 
Compensation dropped. Varying the assumptions underlying the projected 
part of this estimate alters the size of the gain somewhat, but even the 
lowest bound estimate yieldp a substantial gain. 

• Government budgets shoved a net loss from TOPS of $1,3 ig per 
experimental. 

The estimated effects of TOPS on state and federal budg' 3 are pre- 
sented in the bottom panel of Table 3. The key item, of course, is the cost 
of the program, which was more than $2,600 per experimental. This was 
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TABLE 3 



MAINE 

ESTIMATED GAINS AND LOSSES PER EXPERIMENTAL AFTER FIVE YEARS, 
FROM THE PERSPECTIVES OF WELFARE RECIPIENTS AND GOVERNMENT BUDGETS 



Component of Analysis and Perspective 


Es t i ma t e 


We L fe re Recipients 




Ge i ns 

Eernings end Fringe Benefits 

TOPS ALLowences and Support Services 

AFDC Peyments 


$ 3493 
650 
472 


Lo s se s 

Tex Pey ment s 

Non-AFDC Trensfer Payments 

Non-TOPS ALLowences and Support Services 


-753 
-562 
-119 


Net Present VeLue 


3182 


Government 8udgets 




Ge i ns 

Tex Payments 

Non-AFDC Trensfer Peyments and Administration 
Use of Non-TOPS Progrems 


$ 1017 
639 
302 


Losses 

AFDC Peyments end Adm i n i s t re t i on 
TOPS Opereting Costs 

TOPS ALLowences, Sup;^rt Services and Wage Subsidies 


-498 
-1643 
-344 


Ne t Pr esen t VeLue 


- 1"129 



SOURCE: See TebLe 6.10 

NOTES: ALL benefits and costs are estimated for e five-yeer time 
horizon beginning et eppLicetion, and ere expressed in 1985 dollars. 
Beceuse of rounding, detail may not sum to totals* Results include esti- 
metes of projected progrem effects beyond the observetion pariod. The net 
present velue is the sum of ell gains end Losses. 
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partly offset by the substantial use of non-TOPS education and training 
services by the controls, which lowered the TOPS net budgetary investment 
to seme degree. The program generated a substantial increase in fax 
payments because of its sustained earnings impact, but no net savings in 
transfer payments. A net budgetary loss resulted for the five-year period 
covered by the analysis, varying the assumptions underlying the projection 
again alters the size of the estimate but even the highest bound estimate 
yields .1 loss. 

• The net value of TOPS to society as a whole was $2,990 per 
reToJwes ta1 ' r ' £lectS 80b « tant i*l redistribution of 

in addition to these effects on enrollees and governnent budgets, the 
program furnished gains to private employers (through the OJT subsidy) and 
community agencies (because of the labor services provided in the work 
experience assignment). Taking into account the large gain for enrollees, 
the loss to government budgets, and gains for employers and community 
agencies, TOPS produced an overall social gain of almost $3,000 per experi- 
mental, underlying this large net benefit to society is a significant 
redistribution of resources, most notably to a segment of the welfare 
caseload. indeed, these results suggest that, in terms of redistribution 
- one of several program goals ~ TOPS can be viewed as a much more 
efficient means of transferring income to the AFDC group it targeted than 
simply raising their AFDC benefits: A net budgetary cost of about $1,100 
produced an income gain of almost three times that amount and benefited 
employers as well. 

******* 
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As a voluntary, intensive and relatively expensive intervention, TOPS 
differed from most of the other welfare employment initiatives evaluated by 
MDRC since 1981* Any comparisons between the TOPS findings and those from 
other states require great care, however, for a number of reasons* 

In particular, TOPS was a small-scale program restricted to recipients 
who had been on AFDC for at least six months, had limited recent employment 
and were not screened out? unlike the other state initiatives studied, TOPS 
was not intended to serve a broad section of the WIN caseload* In part 
because of the voluntary nature of the program and the screening that 
occurred, the percentage of people in the experimental group who parti- 
cipated in the program was far higher in TOPS than in the other programs 
studied. The impacts in MDRC's studies are averaged over all members of 
the research sample, including both participants and non-participants. 
Since the TOPS impacts are spread over fewer non-participants, it might be 
expected that estimated impacts would be larger than in MDRC's other 
evaluations. Cross-state comparisons are further complicated by variations 
among states in economic conditions and welfare benefit structures, factors 
which can have a strong influence on the nature of the AFDC caseload and 
the outcomes for both experimental s and controls. 
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MAINE: 
FINAL REPORT ON THE 
TRAINING OPPORTUNITIES IN THE 
PRIVATE SECTOR PROGRAM 



0 ^ 



CHAPTER I 



INTRODUCTION 

This is the second and final report on the State of Maine's Training 
Opportunities in the Private Sector (TOPS) Program, ^ops was operated as 
one option among an array of employment-related activities available to 
recipients of Aid to Families with Dependent Children (AFDC) through 
Maine's Work Incentive (WIN) Demonstration system. Thus, the evaluation 
concerns a particular program alternative, in which enrollment was 
voluntary, rather than the entire Maine WIN Demonstration system, 

Maine's WIN Demonstration is called the Welfare Employment, Education 
and Training Program, or WEET. The WEET initiative became possible through 
the federal Qnnibus Budget Reconciliation Act of 1981 (OBRA), which allowed 
states to reorganize the Work Incentive program, the major federally- 
funded employment program for AFDC recipients. 

TOPS was distinguished from other WEET activities by three program ele- 
ments. First, it was a prescribed sequence of activities — pre-vocational 
training, followed by unpaid work experience, followed by on-the-job train- 
ing (OJT) in a subsidized position, preferably in the private sector. 
Second, it was intended to reach a harder- to- employ group among the AFDC 
caseload, as measured by prior welfare dependency and relative lack of work 
experience, although candidates had to apply and go through additional 
screening before being judged appropriate to participate. Third, TOPS 
tended to involve more intensive use of staff time than other WEET program 
activities. 
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In addition to these programmatic differences, TOPS differed from WEET 
in that it used grant diversion to finance part of the wage subsidies 
offered to employers who hired TOPS participants in the OJT component. 
Grant diversion is the financial mechanism through which all or part of a 
public assistance grant is used to finance program services for recipi- 
ents as opposed to payments made directly to participants. Authorization 
for states to use grant diversion to fund OJT programs for AFDC recipients 
was in^uded in OBRA. 

TOPS was operated as an experimental program, beginning in October 
1983, by the Division of Welfare Employment, Maine Department of Human 
Services.* The prescribed sequence was available statewide, at different 
times at dif f ere.:*: locations. Program intake ended in December 1984, 
although the operational phase continued into 1986, as enrollees finished 
their OJT positions. Since December 1984, the TOPS sequence has not been 
offered as a regular part of WEET, but similar omponents are available to 
WEE? registrants on an individual basis. 

Research on TOPS was conducted by the Manpower Demonstration Research 
Corporation as part of its 11-state Demonstration of state Work/Welfare 
Initiatives. An earlier study of TOPS, published by MDRC in June 1985, 
reported on the implementation of the program through September 1984. 1 
This report examines the implementation experience of the whole demonstra- 



* In December 1987, the Division of Welfara Employment became the Division 
of Family Independence within the Bureau of income Maintenance, Department 
of Human Services. 
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tion period. It also analyzes the impacts and cost-effectiveness of the 
TOPS sequence. 

I. Goals and Intentions 

TOPS grew out of work initiated by a committee appointed in late 1980 
by the Commissioner of Human Services to develop recommendations for improv- 
ing Maine 1 s employment and training services for AFDC recipients. 2 In 
addition to the grant diversion option, the passage of OBRA offered states 
increased flexibility in the administration and content of welfare employ- 
ment programs. This provided the opportunity for the state to implement 
many of the committee's recommendations in «.he WEET program, which replaced 
Maine's regular: WIN program in April 1982. Plans for operating TOPS ai, an 
experimental program within WEET began in the late spring of 1982. 

The Division of Welfare Employment had multiple goals in mounting the 
TOPS demonstration. First, officials wanted to work with a subset of the 
AFDC caseload which they felt had traditionally been underserved in Maine's 
training programs — women with long-term welfare dependence and sporadic 
work histories. Second, they wanted to move these women into jobs that 
paid better than the minimum wage and offered opportunities for advance- 
ment. The assumption was that the prescribed combination of services, 
designed to address problems arising from AFDC recipients 1 low self-esteem 
as well as their lack of work experience, would help the women to obtain 
higher-paying jobs enabling them to become less dependent on welfare. This 
was of particular concern in Maine wher*, historically, a relatively large 
proportion of the caseload worked but continued to receive benefits.^ 
Although it was anticipated that welfare savings would result, reductions 
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in welfare payments were not che primary goal, staff were more interested 
in improving women's economic status by helping them obtain better-paying 
jobs. Third, staff wanted to experiment with grant diversion. They 
considered it a potentially more productive use of resources than direct 
cash payments. They also viewed it as a new source of funding in an era of 
shrinking federal appropriations. instead of coming out of general WEET 
funds, the employer subsidy in TOPS OJT positions could be covered in part 
by funds available for AFDC benefit payments. 

n * Key Featu res of the tops Program Model 

This section describes the key features of TOPS in more detail. 
A. Eligibility Criteria 

To be considered for TOPS, AFDC recipients had to apply. Not everyone 
who applied, however, was accepted. The targeting strategy used in TOPS 
combined two different aproaches that were somewhat contradictory in 
intent. 

On the one hand, guidelines reflected the interest in working with the 
harder-tc-employ by specifying objective eligibility based on length of 
time on welfare and employment status. TOPS enrollment was limited to 
single heads-of-household who had been receiving AFDC for six consecutive 
months, and who were not employed at the time of enrollment. On the ether 
hand, staff were supposed to screen out eligible women who had child-care, 
transportation, health or other problems that could interfere with their 
participation, as well as women who were unable to read at the level of the 
materials used in the pre-vocational training classes. In practice, staff 
developed additional, informal screening criteria and sought to enroll 



eligibles who could demonstrate high motivation, had a high school diploma 
or GED, and whose job goals were commensurate with their current skills. 
These screening efforts in fact worked against the goal of targeting the 
program to the harder- to- employ. 

in addition, recruitment efforts frequently focused on wanen who were 
exempt fran the WIN registration requirement because they hud children 
under the age of six or lived outside a mandatory service area. As a 
result, 61 percent of the research sample was composed of WIN-exempt wanen 
not necessarily typical of the WIN caseload. 

B. The TOPS Sequence 

The TOPS sequence had three phases, to be undertaken in the following 
order: 

1» Phase I; Pre-vocational Training 

The pre-vocational training component lasted two to five weeks 
and stressed personal growth as well as the development of job-seeking and 
job-holding skills* Goal-setting and decision-making, household budgeting, 
and balancing the denanOs of family and work were also addressed. Bnphasis 
was placed on career exploration; fostering good work habits and attitudes 
such as dependability, punctuality and responsibility; and teaching the 
mechanics of filling out resumes, conducting a self-directed job search and 
improving interviewing skills. Participants continued to receive their 
AFDC grants while taking part in this activity and were paid an allowance 
of $1.25 per hour to cover training-related expenses. Basic education was 
also available, on a limited basis, for clients who sought it or wfcre 
thought to need it during Phase I and Phase II. 
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2. Phase II: Work Experience 

At the end of phase I, participants were reassessed to determine 
whether they needed further training or were ready for OJT positions in the 
private sector. Participants who required additional assistance were 
placed in a 20-hour per week, unpaid work experience position in the public 
or non-profit sector, for up to 12 weeks. 4 These work experience assign- 
ments were intended to teach or reinforce good work habits, and to provide 
participants with an employer reference. Participants continued to receive 
their grants, were paid for training-related expenses, and were eligible 
for basic education. 

3. Phase III; Oi>the-Job Training 

TOPS . participants who demonstrated their motivation and 
acquisition of basic work skills became candidates for placement into 
on-the-job training (OJT) positions, preferably in the private sector, in 
an OJT program, employers receive a subsidy for part of the wages paid to a 
new worker during a specified training period, and ar>> expected to retain 
the worker as .a permanent employee at the end of thet period. One 
rationale for an OJT subsidy is that it encourages employers to hire 
workers who would otherwise have difficulty obtaining such jobs. Another 
is that the subsidy compensates employers for lower productivity during th 
training period or is a reimbursement for the training provided. In TOPS, 
the training period was limited to a maximum of six months and the employer 
subsidy was set at 50 percent of the new employee's wages. 

The WEET staff who designed TOPS assumed that not all participants 
would need unpaid work experience before obtaining an OJT position? the 
design thus allowed clients who were judged to be ."job ready" after pre- 
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vocational training to move directly from Phase I to Phase 111. These two 
phases were considered the primary components of the intervention strategy. 
In actuality, however, a higher proportion of enrollees participated in 
Phases I and n than in any other combination of components. 
c * Grant Diversion; The Funding Mechanism for OJT 

in TOPS, as noted, the employer subsidy in the OJT phase was funded in 
part through the use of grant diversion. The mechanism for grant diversion 
worked as follows: As in any OJT program, TOPS participants were placed in 
jobs with employers who had been promised a wage subsidy. As federal 
regulations require, a portion of the recipients' wages wis counted as 
income that served to reduce the amount of the recipients' welfare grant. 
Normally, the amount by which the grant is reduced would be an immediate 
savings of public expenditures; under grant diversion, this amount is not 
saved, at least in the short run. it is deposited into a pool of funds 
along with the grant amounts diverted for the other OJT participants. Wage 
subsidy payments to employers are then made from the pool as required by 
the OJT placements. The advantage of using a pool to finance the subsidies 
is that the amount an employer receives does not have to be linked directly 
to an individual grant. 

Grant diversion is intended to be an investment. Budget savings from 
the reduction in the welfare grant are foregone in the short-term in favor 
of investment in wage subsidies that are expected to provide AFDC recipi- 
ent .,-ith direct access to stable employment,, particularly in higher- 
paying jobs in the primary labor market. The implicit assumption is that 
there will be long-run savings as participants become more economically 
self-sufficient and remain off welfare. 
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It should be stressed that grant diversion was merely one of the 
funding mechanisms for the OJT component of TOPS ; the program treatment 
that is being tested in the demonstration is the threa-component sequence 
that constitutes TOPS. Under grant diversion, the dollar amount of the 
grant paid directly to recipients was lowered during the OJT employment in 
exactly the same way as it would be for any recipient who works. But, 
because the amount by which the grant is reduced Is used to offset the OJT 
cost, it cannot, by definition, generate welfare savings during the program 
period. 

111 • Overview of .Program Administratio n 

Since the. TOPS demonstration was implemented as one option among a 
comprehensive array of services available through WEET, it is important to 
understand the WEET system of which it was a part. It is also important to 
understand the interagency coordination necessary to staff and implement 
the full TOPS sequence. 

A « Maine's WIN Demonstration Program 

The WEET program replaced regular WIN in Maine in April 1982; the 
change, which followed a 30 percent reduction in federal funding, 
occasioned a major administrative reorganization and a reorientation of 
program services. The regular WIN program had been operated jointly by the 
Maine Employment Service and the Separate Administrative Unit within the 
Department of Human Services. WEST, in contrast, was administered solely 
by the Division of welfare Employment, a newly-created office within the 
Department of Human Services. Faced with the reduction in federal funding 
— which was only partially offset by additional state monies -- the 
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central WEET staff closed two out of the seven WIN regional offices in the 
state, scaled back the areas where AFDC recipients were required to 
register for employment services, and eliminated over half the WIN staff 
positions. The WEET offices were staffed with a combination of former 
Employment Service and Separate Administrative Unit staff, now located in 
the same offices for the first time. 

The transition from WIN to WEET also signalled a change in administra- 
tive styles and service priorities. WIN utilized specialized positions in 
such areas as assessment, job placement and support services, while WEET 
workers are expected to be generalists with case-management responsibility 
for all aspects of registrants 1 needs. In contrast with WIN'S emphasis on 
quick placements, WEET places a premium on providing opportunities for 
longer-term education and training and more support services. WEET staff 
provide few direct services other than assessment, the preparation of 
employability development plans and job search activities. In addition, 
they are expected to inform clients about ->ther programs and services 
available throughout the community, make referrals or direct arrangements 
for such services, and oversee the clients 1 progress. 

Statistics on WEET registrants' activities during the TOPS demonstra- 
tion period indicate the kinds of alternative services that would be 
available to WEET enrollees in the absence of TOPS. They also reveal the 
relative importance of education and training in the menu of WEET-funded 
activities. As of September 30, 1983 — a few weeks before the start of 
the TOPS demonstration — WEET had a caseload of .3,157 heads-of-household. 
Of these, 700 women were enrolled in training or education programs? and 
334 were in job search activities. Over the course of the next year, 
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4,146 WEET registrants went through assessment, 1,611 participated in 
training or education activities, and 703 were active in job search or job 
development. 5 

B » Interagency Coordination 

Central WEET staff recognized that the system hafi neither the financ- 
ial resources nor the staff capacity to operate the full TOPS sequence. 
Therefore, they contracted with the state's major training system for the 
disadvantaged - authorized under the Job Training Partnership Act (JTPA) 
- to nrovide most of the Phase I and some of the Phase III services and to 
help underwrite the OJT subsidy cost. The following division of labor 
applied to most areas in the state: WEET staff were responsible for recruit- 
ing and screening all prospective TOPS enrollees; JTPA staff operated the 
pre-vocational training workshops; WEET was in charge of developing and 
monitoring the work experience positions; JTPA, either elone or in conjunc- 
tion with WEET, was responsible for developing and placing participants in 
OJT positions. (For details, see Table 1.1.) Financial agreements 
specified that WEET would pay 70 percent, and JTPA 30 percent, of the wage 
subsidy provided to the OJT employers. Only the WEET share was funded 
through grant diversion. 

Between October i 982 when the Job Training Partnership Act was passed, 
and October 1983 when it replaced the Comprehensive Employment and Training 
Act, Maine's employment and training system for the disadvantaged went 
through a restructuring similar to that ' experienced in the welfare employ- 
ment system, m the JTPA system, as in WEET, funding was reduced, adminis- 
trative position- were reorganized, staff positions were eliminated or 
restructured, and specialist f-nctions were transformed into generalist 




-10- 



4 



*3 



TABLE 1.1 



MAINE 



WEET AND JTPA AQENCY CONTRACTUAL A3REEMENTS 
K?> TOPS SBNICES, BY WEET RB3ION 



I 

H 
H 
I 



WEET Region 


TOPS Slte/WEET Office 


JTPA Service Provided 


WEET/JTPA Contrectual Agreement 


I 


e 

Portland 
Blddeford 


Training Reeource Center 
Heine Department of Labor 


n »i o r h ai a recruitment and eeecBBjnont, 
Pre- Vocational Training, end Work Experience. WEET 
end TRC ehared OJT development 

15-county errengement d 


II 


B 

Lewi aton 
Norway 


Mountain Volley Training 
Mounteln Valley Trelnlng 


15-county arrengoment 
15-county arrangement 


III 


Auguete 
Bath 

Damarl ecotte 

Skowhayan 
Weterv ULa 


Maine Department of Labor 

Coastal Economic Development 
Corporation 

Coeetel Economic Davelopme* : 
Corpora tl on 

Maine Department or Lebor 

Hai^ Department of Labor 


15-county arrangement 

WEET did recruitment* JTPA hih nnoi naoaaonant 
end provided ell other TOPS aorvicee 

Same contract ea Beth 

15-county arrangement 
15-county arrangement 


IV 


_ a 
Bangor 

Ellaworth 
Hachl aa 


Trelnlng Development Corporetlon 
Training Development Corporation 

Ma \ nn Do nn t* trt o n t* t nknn 
naiiitJ vupcsi UwUllb Of LBDOr 


15-county arrangement, except WEET end TDC ehared 
OJT development 

Same contract aa Bangor 

15-county arrangement 


V 


Praaque late 8 

Caribou j 

Houltgn f 
- — * 


Arooetook County Action Program 

Arooetook County Action Program 
Arooetook County Action Program 

r 


15-county arrangement, except WEET provided Pre- 
Vocatlonal Training 

Same contract ea Preaqua Iele 

Same contract ee Preequ- Iel9 
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SOURCE: HDRC interviewe with TOPS progran eteff and WEET/JTPA contracta. 

NOTES: "indlcatee WEET regional offlca. 
b 

the ".unba-lend Count"- MA ^ ° f *! T0RS d8M0n8t <- 8 t<°n '""re wore two Servloa Delivery Areoe (SOAe) In Heine. In 

fenrtaant of t2S?SLJiS I?,"*" 0 " BB0 " r ™ C * ntBr 89 ™ 8d 88 °"h 'he 9'ent recipient end the aarvlca provider. The Maine 
D°£r£?nt ' &»ploy«ont end Training, adal nl etered JTPA prograae In the 15-county SOA. In threi counties! the 

prZTr ege P ncTee! MrV>C8e dlractl *'' 1n th8 «- 88t * SDA, it eubcontrected aarvlca delivery with e verlety of aa^lca 

c 

In the November 1383 cycle only, WEET staff did the recruitment end eeeosanent. 

d 

Training and aJT^JoloSnu' 5 ' BrranoeB,ent ' " EET did recruitment, eeeee*,ent. end Work Exparlence; JTPA provided Pre-Voceti onal 
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roles. The repercussions of these changes were still being felt when the 
TOPS demonstration began. 

c » The Discontinuation of TOPS 

intake for the TOPS demonstration ended in December 1984. Since then, 
the Division of Welfare Employment has not offered the TOPS components as a 
sequent program. WEET continues to place individuals in pre-employment 
training offered by JT PA or other providers, but this is not required to be 
in combination with the other components that made up TOPS, and the WEET 
registrants are mainstreamed with non-AFDC recipients. Work experience and 
OJT positions are also available through WEET without being part of a pres- 
cribed sequence, and OJT continues to be funded through grant diversion. 
Coordination efforts with JTPA have continued, but on a largely informal 
rather than contractual basis. 

The Division of Welfare Employment discontinued the TOPS sequence 
after the special funding from the federal Office of Family Assistance 
(OFA) ran out. m addition to the funding constraints, the decision not to 
institutionalize TOPS reflects opposition to the demonstration among sane 
local staff who objected to the prescriptive nature of the program model, 
argued that it was too time-consuming and costly to implement, and doubted 
that it was more effective than other WEET services. 

IV. The Evaluation Design: An Overview 

MDRC's evaluation of TOPS examines the experiences of 444 female 
heads-of-household enrolled ln the research sample between October 1983 and 
December 1984. 

The evaluation uses a random assignment experimental design, the most 
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powerful methodology for producing reliable evidence of program effects* 
AFDC recipients who applied to participate in TOPS, met the eligibility cri- 
teria, and were deemed /appropriate; by TOPS staff were randomly assigned 
to either an experimental or a control group. Two experimental were 
assigned for every one control. Members of the experimental group were 
eligible for the full TOPS sequence, but not for other education and train- 
ing activities except limited basic education* Members of the control 
group vrere to be treated like other WEET registrants: they were eligible to 
enroll in any employment- or education-related activity available through 
WEET, JTPA, community colleges or other organizations. This included 
participation in components similar to those offered in TOPS — pre-voca- 
tional training, work experience or QJT — but not in all three. Nor was 
there any attempt to coordinate a sequence of components* Members of the 
control group had less contact with staff, and any pre-vocational training 
they received was not in the same classes as those held for the TOPS 
sequence. 

The behavior of the control group measures what the experimental group 
would have experienced in the absence of TOPS* Because individuals had to 
volunteer to participate in TOPS and were screened before random assignment 
for problems that could interfere with participation, it was likely that 
members of tfr'- control group would seek out and receive WEET or other alter- 
native services. This did in fact happen: 52 percent of the controls 
participated in employment-related activities* This means that the experi- 
mental-control differ en^cs in the Maine evaluation are a measure of the 
effectiveness of offering the TOPS sequence compared to a range of alter- 
native activities available through WEET or other community programs, 
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including seme components similar to those offered in .TOPS. 

The evaluation has three parts: an impact analysis, a benefit-cost 
analysis and a process analysis. The impact analysis measures the effects 
of the TOPS sequence on the employment, earnings and welfare receipt of 
enrollees. Outcomes in these areas are analyzed using information from a 
variety of state records, including the earnings reported to the Unemploy- 
ment insurance system, AFDC payment records and WEET activity data relating 
to TOPS and other services. 

The benefit-cost analysis uses the impact estimates and a wide variety 
of other sources to take account of the program's benefits and costs as 
comprehensively as possible. These benefits and costs are assessed from 
several different perspectives — in particular those of government 
budgets, taxpayers, welfare recipients and society as a whole — to allow 
estimates not only of the efficiency of the program but also of the gains 
and losses to the groups it affects directly. As in the impact analysis, 
the benefits and costs are estimated as the net effects of TOPS (i.e., the 
differences between the benefits and costs for the experimental group and 
those for the control group). 

The process analysis describes the urogram in operation and was the 
focus of the interim study. m addition to discussing management issues, 
particularly interagency relationships in the implementation of TOJS, the 
interim study examined the recruitment and participation patterns of the 
research sample randomly assigned through March 1984 and the process of OJT 
development. The process sections of this final report prtwnt data on 
program participation that were unavailable at the time of the earlier 
report. They focus in particular on participation in the OJT component and 



its function as a transitional employment strategy. 



v# Differences Between the TOPS Evaluation and Other MDRC Evaluations 

The TOPS evaluation differs in a number of important ways from other 
evaluations of welfare initiatives conducted by MDRC since 1981. 6 First, 
in other studies, MDRC evaluated a large part of the WIN or WIN Demonstra- 
tion system; in Muine, it evaluated TOPS, a single program offered within 
the WIN Demonstration system (WEST). Second, the other initiatives were 
broadly targeted to the WIN-mandatory applicant and/or recipient caseload, 
whereas Maine targeted TOPS to a small subset of the AFDC caselo*. and re- 
cruited voluntary WEET registrants - well as mandatory registrants. 
Third, the segment of the caseload eligible for TOPS was screened for 
interest, appropriateness and potential barriers to participation before 
being randomly assigned, a practice not follcved in other states. Fourth, 
no one war, required to enroll m TOPS — the research sample was composed 
of eligible individuals \ho asked to participate. Thus, the impacts pre- 
sented for Maine apply only to a very small proportion of the AFDC 
caseload, not to the substantial portion seen in other states. Finally, 
the high level of participation by the control -*-oup in employment *nd 
training activities differentiates the TOPS evaluation fran others in 
MDRC's recent evaluations of welfare employment initiatives. 

In terms of the scale and voluntary nature of the program, it is 
probably more appropriate co think of the TOPS evaluation in the context of 
the National Supported Work Demonstration, designed and managed by MDRC 
between 1974 and 197 9, rather than MDRC's recent evaluations of welfare 
employment initiatives. 7 Like TOPS, Supported Work was a relatively small- 



scale program. It included among its target groups AFDC recipients who had 
beer on welfare for at least 30 out of the prior 36 months and who had 
applied to participate. Participants were offered subsidized jobs — paid 
for, in part, through grant diversion — designed to develop basic employa- 
bili';y by providing close supervision, peer support and graduated stress. 

VI. Policy Relevance of the Demonstration 

A number of features make the TOPS demonstration of particular rele- 
vance to policymakers and program administrators. These include: the 
nature of the TOPS targeting strategy; the nature of the services provided; 
the grant diversion funding mechanism; and ±e challenge posed by 
interagency coordination. 

The TOPS targeting strategy provides important lessons about how 
• disadvantaged program participants are likely to be if states seek to 
enroll the harder-to- employ as measured by length of welfare receipt and 
prior employment history but at the same time give priority to those who 
volunteer for services. This is especially true if staff are allowed 
discretion in deciding who among the technically eligible is •approjpriate..' 
Targeting the harde;:-tc-eaploy but giving priority to those who apply is an 
approach that figures prominently in some recent legislative proposals for 
reforming the nation's velfare system. Those who a^ply for services and 
are accepted by the program are likely to be more motivated or face fewer 
barriers to participation than other WII registrants. This self- select ion 
factor is extremely important i.n understanding the behavior of the TOPS 
research sample, because 61 percent of the TOPS sample ( experiment als and 
controls) were not even required to register for WIN, primarily sinoe they 



had children under the age of six. In addition, although TOPS sample 
members were disadvantaged in that they had prior welfare experience and 
little prior employment, they were relatively well-educated. 

Many questions remain about the relative effectiveness of providing 
more intensive services than job search or work experience to the AFDC 
caseload. The TOPS sequence qualifies as an intensive program by virtue of 
its length (it typically took 11 months to finish the full cycle); the 
amount of staff time devoted to the provision of services? and its cost. 
The average cost of operating the TOPS sequence for each menber of the 
experimental group was $2,627, including the costs of support services and 
employer wage subsidies. An additional $134 was spent per experimental on 
education, job. search and training activitier not available through TOPS 
and accompanying support services. By comparison, an average of $517 was 
spent per control in the TOPS research sample. The demonstration provides 
an opportunity to learn about the impact** and cost-effectiveness of the 
particular type of treatment offered in TOPS for a selected group that is 
not typical of the AFDC population as a whole. It should also be stressed 
that TOPS <iid not provide classroom training or extensive ^pportu- * ties for 
remedial or basic education — services that also fall into the intensive 
category. 

The grant diversion funding mechanisn in the OJT component makes TOPS 
f considerable interest to policymakers, as questions renain about the 
possible scale, cost implications, and administrative feasibility of this 
investment strategy when used to fund an OJT program. To answer sane of 
these questions and encorrage plates to use grant diversion, the federal 
Office of Family Assistance made available, on a competitive basis, special 
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funds to operate and evaluate grant diversion programs in 1983. Maine was 
one of six states selected for such an award. These funds, together with a 
matching grant from the Ford Foundation, made the TOPS evaluation 
possible. 8 

Finally, the emphasis on interagency coordination in operating TOPS 
provides lessons about the obstacles to effective coordination between 
welfare employment initiatives and jtpa programs, and possible ways to 
overcome them. These issues were discussed in detail in the previous 
implementation report. 



* * * * * 



The remainder of this report consists of six chapters. Chapter II 
describes the research design and data sources, the recruitment and 
screening process, and the characteristics of the research sample, chapter 
III discusses the participation patterns of the experimental group in TOPS 
and the implementation of each of the components that make up the TOPS 
sequence. chapter IV examines the behavior of the control group and dis- 
cusses labor market conditions and AFDC grant calculation in Maine during 
the demonstration period to provide the basic context for interpreting the 
impact and benefit-cost findings that follow. chapter V considers the 
program's impacts on employment, earnings, and welfare receipt. Chapter VI 
weighs the costs and benefits of the program from the perspectives of 
welfare recipients, government, taxpayers and society as a whole. Chapter 
VII concludes the report with a discussion of the broader implications of 
the TOPS experience. 



CHAPTER II 



RESEARCH SAMPLE, DESIGN AND DATA SOURCES 

This chapter begins with a description of the procc-ss of selecting the 
research sample and a presentation of key characteristics of the TOPS 
sample members in comparison to those of the entire WEET caseload. It then 
discusses the experimental design used in the evaluation. It concludes by 
outlining ";he key data sources and assessing their accuracy. 

I. The Research Sample 

A. Recruitment and Assessment 

In contrast to other welfare employment initiatives recently studied 
by MDRC, TOPS was never designed to get as many clients as possible into 
the program. Instead, it was intended to be implemented as one option 
among an array of activities available to WEET registrants, and to serve a 
relatively small subset of the caseload. AFDC recipients were not auto- 
matically assigned to TOPS; they had to apply to participate, and meet both 
objective and subjective eligibility criteria before being enrolled. 

Recruitment and assessment was the responsibility of the WEET staff, 
although in sane locations JTPA staff sat in on the assessment interviews. 1 
To be eligible for TOPS, wanen had to have been on welfare for at least six 
months and not ba currently employed. In addition, they were to be 
screened out if staff felt they were not /appropriate; candidates because 
problems with child care, transportation, emotional or physical health 
presented barriers to participation, or they lacked sufficient motivation 
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or could not pass a basic reading test. In practice, local staff exercised 
cons/ Arable discretion in deciding whether a potential client was 
appropriate for TOPS, going beyond these criteria. sane staff were 
reluctant to work with women who lacked a high school diploma or GED, or 
had only minimal occupational skills because of their concern about putting 
unprepared clients into OJT positions. Concerned also sbout the motivation 
of participants, staff weighed the reasons people gave for their interest 
and sometimes looked for additional proof of motivation or character 
reccoeendations. 

While sane WEET staff were willing to ."take a chance.' on applicants 
they had doubts about, others - influenced in part by concerns expressed 
by JTPA staff ^- preferred not to meet their enrollment goals rather than 
to enroll more questionable applicants, staff records indicated that, for 
the sample randomly assigned between October 1983 and March 1984, about 
three out of every ten women who applied and were interviewed for TOPS were 
considered inappropriate. 

The care WEET stair* took in selecting women for TOPS helped keep the 
number of enrollees relatively small and turned recruitment into a 
protracted process. in every officr, staff identified recruiting and 
interviewing applicants as the most time-consuming task in implementing 
TOPS. Seventeen percent of the total TOPS operating cost of $479,003 was 
spent on this process. 

The outreach and assessment process involved two steps. First, staff 
reviewed their WEET files to identify potential eligibles already in the 
caseload, and contacted those individuals by phone or letter. In one 
location, staff also made home visits. second, to identify potential 
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eligibles not currently registered with WEET, staff reviewed lists of 
welfare recipients provided by the Bureau of Income Maintenance of the 
Department of Human Services and mailed letters to possible candidates. 
Advertisements were also placed in newspapers some weeks before the start 
of eacfc cycle. Interested applicants were a^ked to call the office to 
schedule an interview; the interviews generally lasted an hour, and in seme 
locations were conducted Jointly by WEET and JTPA staff. 
B. Research Design and the Structure of TOPS 

After screening, all the selected enrollees were randomly assigned to 
either experimental or control status. The inclusion of a control group 
was intended to permit the estimation of net program effects by comparing 
measured outcomes (such as earnings and welfare receipt) of the experi- 
mental group with those of the control group. Research has shown that a 
substantial proportion of AFJC recipients find jobs and leave welfare on 
their cwn, without receiving services. 2 A control group makes it possible 
to separate out this movement (which is a product of individual character- 
istics of the recipients and factors in the wider environment) from the 
effects of the program, by contrasting the outcomes for two research groups 
who are similar in all respects except the program treatment. 

Experimental^ were to enroll in the TOPS program with its prescribed 
sequence of prevocational training, unpaid work experience, and then OJT. 
Controls could receive any WEET services — including prevocational train- 
ing, unpaid work experience, or OJT — but not the entire TOPS sequence of 
activities, ( See Figure 2 .1 . ) The control group, therefore, does not 
represent a •no-program" benchmark but rather a benchmark of alternative 
program services. Chapter IV describes the experiences of the TOPS 
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FIGURE 2 . 1 
MAINE 

TOPS RESEARCH DESIGN 



AI AFDC Recipients 
who Appfed 
for TOPS 



ElgiWe for Regular 
Services AvaiaWe 
through WEET a 




r 



Controls 



Experimentals 



Efigible for Regular 
Services Avaiabte 
through WEET 
(inducing: 

Pre-Vocational TraWng, 
Work Experience, or OJT, 
but Not the TOPS Sequence) 



Pre-Vocational 
Training* 3 



Wort 
Experience b 



On-the-Job 
Training 



\J\ The TOPS 
i / Sequence 



b ?* r69U u r ! efVi ° eS aVa3able thr0U * ^ m the 3311,6 servicesavaiable to the control group 
m some locations experimentals were allowed to participate h Emited basic education . 
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evaluation's control group in detail. 

Because of the relatively small size of the WEET registrant pool in 
Maine and the selectivity of the recruitment process, randan assignment was 
conducted at a ratio of two experimentals to one control. WEET program 
operators felt that it was important to serve a large proportion of those 
recruited? a research design in which only 50 percent of the sample 
received services was not considered acceptable. This larger proportion of 
experimentals also ensured that sufficient numbers participated in the 
third and final component (on-the-job training) to permit discussion of 
this component. 

C. Sample Build-up 

The random, assignment period lasted from October 18, 1983, to December 
13, 1984, during which 297 clients were randomly assigned to the experi- 
mental group and 147 to the control group. 3 The multi- component sequence 
of activities that constituted TOPS was not operated as a continuous 
on-going program that eligibles could enter at any time. Rather, TOPS was 
offered in a series of discrete cycles, begun at irregular intervals, each 
consisting of the three components. The start of a pre-vocational training 
class was considered the start of a cycle and the wcmen who began 
pre-vocational classes together were treated by staff as a distinct croup, 
even though they moved through the later components in the sequence at 
varied paces* Over the course of the demonstration, 30 TOPS cycles were 
operated at 15 different locations throughout the state; the first cycle 
began in November 1983, the last in January 1985. (See Table 2.1.) 

The sample build-up in each region over the enrollment period is shown 
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TAB I E 2,1 
MAINE 

TOPS PROGRAM CYCLES, BY SITE 



R ft r i ft n 


Numb e r 
of Cycles 


Renge of Perticipents 
per Cycle 


Region I 






Port lan 1 


4 


6-21 
6-10 


Biddeford 


3 


Region II 






L e w i s t o n 
Norway 


3 
2 


6-10 
4-7 


Region III 






Auoustfl 

*^ U H U w w O 


1 


S 


Ba th 

Darner iscotte 


2 
1 


9-10 

5 


Skowh egan 


1 


5 
9 


We te rv i 1 1 a 


1 


Region IV 






Bangor 
Ell swor th 
Me ch i a s 


4 
2 
1 


5-10 
4 

7 


Region V 






Presque Isle 

Caribou 

Ho u I ton 


2 
2 
1 


8-9 
12 - 16 
13 


Total 


30 





SOURCE : Maine Central WEET office calculations from TOPS Client Tracking 
Reports. a 



NOTES: Participation is defined as attending Pre-Voca t i one I Training for 
at least one day. a 
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in Table 2.2. The target sample size was 900 experiment als and controls, 
much larger than the actual sample of 444. This shortfall reflected 
problems in making eligible women interested in applying fot* the TOPS 
combination of services in preference to other WEET activities, as well as 
staff selectivity in enrolling clients. The small sample size, therefore, 
should not be taken as an indication of any general lack of interest in 
receiving services on the part of AFDC recipients. Indeed, about 50 
percent of new WEET registrants are AFDC recipients who are not mandatory 
WIN registrants and therefore do not have to register for any employment 
services.* 

All the TOPS sample members were female single heads-of-household. 
Nearly all vers white and over half were divorcer or widowed. In keeping 
with the program goals, most jf the sample members were relatively long- 
term AFDC recipients ~ 63 percent had been receiving AFDC for more than 
two years — and only about one-third had been employed in the year prior 
to random assignment. Half (49 percent) had children under the age of six 
and 61 percent were exempt from WIN registration requirements. The sample 
members had worked an average of only three months in the two years prior 
to random assignment. However, due to the additional screening criteria 
developed by local staff, they were relatively well educated: 75 percent 
had received a high school diploma or GED by the point of randan 
assignment. (For sample characteristics see Tatle 2.3.) 

Random assignment was largely successful in producing comparable 
experimental and control groups. (See the first two columns of Table 2.3.) 
However, as expected when dealing with small sample sizes, there were some 
differences between the experimental and control groups. Statistically 



TABLE 2.2 



MAINE 



DISTRIBUTION OF SAMPLE MEMBERS, 
8Y PERIOD OF RANDOM ASSIGNMENT, REGIONAL OFFICE, AND RESEARCH GROUP 



Period 
of Random 
Assignment 


Region I 


Region II 


Region III 


Region IV 


Region V 


Totel 


Experi- 
mented 


Contro' s 


Experi- 
mented 


Controls 


Experi- 
mented 


Controls 


Experi- 
mented 


I 

Controle 


Experi- 
mented 


Controls 


Experi- 
mented 


I 

i 

Controle 


October- 
Oecember 1983 


34 


16 


10 


5 


13 


7 


21 


8 


8 


5 


87 


42 


January- 

Lit* nnh 4 OOA 

Keren iub<* 


18 


9 


8 


4 


25 


10 


18 


10 


22 


14 


92 


47 


April- 
June 1884 


8 


4 


12 


8 


3 


0 


10 


3 


18 


8 


52 


24 


July- 

Septumber 

1884 


8 


4 


8 


2 


6 


4 


4 


2 


10 


4 


36 


16 


October- 
December 1984 


6 


3 


4 


3 


4 


3 


3 


2 


13 


7 


3D 


18 


Total 


74 


36 


42 


22 


51 


24 


57 


26 


73 


38 


297 


147 



SOURCE: Tabulations fro* MORC Cliant Information Sheets. 
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TABLE 2.3 



• MAINE 

SELECTED CHARACTERISTICS OF THE RESEARCH SAMPLE 
AT THE TIME OF RANDOM ASSIGNMENT, BY RESEARCH GROUP 



on oroctBri oui c 


Experi mentals 


Controls 


Total 


Region (X) 








I 


24.9 


24.5 


24.8 


II 


14.1 


15.0 


14.4 


III 


17.2 


16.3 


16.9 


TV 


4 Q O 

1 9 • c 


4 7 7 


4 Q 7 
1 O . / 


V 


OA R 


O Q R 
CO i w 


c O . c 


Ann f % 1 








24 Years or Loss 


25.1 


17. P 


22.7 


25 to 34 Yee rs 


50.5 


47. S 


49.4 


3 C tn 44 Ynn pc 




QO 4 
CO • 1 


OO 0 * 
C C a a 


45 to RQ Yoo pr 


4 1 




s n 


Average Age (Yeersj 


29.6 


31.1 


30.1*» 


tthnicity (X) 








Wh i te, Non-Hispanic 


96.9 


100.0 


97.9* 


oucK| non**n i BpB n i c 


4 i 


r n 


4 O 

1,2 f 


n i o pun i c 


n 7 


n n 


n k 
0.5 


Other 


n 7 


n n 
U • u 


U . O 

; 


Degree Received [%) 








None 


26.0 


22.9 


25.0 


ua n □ r b l cq u 1 valency u i p u om a 


co • b 


17 A 

27 . 1 


24.8 


High School Diploma 


50.3 


50.0 


50.2 


Mvoruyti n i y 1 1 a a i« vj i a u a u Utn pictoo 


4 4 O 
11 I J 


11 i J 


4 4 O 
1 1 . O 


Marital Status (X) 








Never Me r r 1 ad 


26.8 


22.9 


25.5 


Harried, Living With Spouee 


0.3 


0.7 


0.5 


Married, Not Living With Spouse 


14.6 


11.4 


13.6 


Divorced or Widowed 


58.2 


65.0 


60.4 


Averege Number of Children 








Under 19 Yee re 


1.79 


1 .84 


1.81 


Any Children (X) 8 








Less Than 6 Yee rs 


52.8 


42.1 


49.3*» 


Between 6 end 18 Yee rs 


64.6 


75.0 


68 0** 



( con t i n uea ) 
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TABLE 2,3 (contl nued] 



Charact er1 stl c 


Expsrl mental s 


Con t r o I e 


Totel 


Prior AFOD Dependency ( % ) 
Two Years or Less 

Un n > Than X m rs V n a m #■> 

naro inan iwo Tears 


35.9 
64*1 


38.8 
61.2 


36.9 
63.1 


Average Konthe on AFDC During 
Two Years Prior to Rendon 


19.4 


19.3 


19.4 


W£ET Stetus (X] 

nanus w u r y 

Vol un to ry 


36 . 1 
83.9 


43.5 
56.5 


38.6 
61 .4 


Avsraga Nunber of Hontha Since 
Most Rscent WEET Registration* 


7.4 


7.3 


7.4 


Average Months Unable to Work 
Due to Medical Problame During 
Two Yeere Prior tc Rendon 

Afisl on a o n t 


1 .4 


1.3 


1.3 


Jeld e Job at Any T1ma During 
Quarter Prior to Rondos 

Aflfil oniflnt f 3M 

no • 1 yilHOIII* I A ) 


15*5 


17.7 


16.2 


Held a Job at Any T1oe During 
Four Querters Prior to Random 

As mi anm «n t f X 1 C 


32 • 7 


38.8 


34.7 


Average Eernlngp During 

Four Quartare Prior to Random 

Anci nnnnnf fSl^* 


447 .51 


682.96 


525.46* 


Average Months Employed During 
Two Years Prloi to Rendora 
Aas1 gnoent 


2.8 


3.8 


3.2* 


For Longest Job Held 1n Paet 

Tw o „ Yea r s 

Avorege Hourly Wege Rata ($) 
Average Weekly Houre 
Duration of Jobe (Months] 


"•01 

30.9 
12.2 


o on 

0.59 
29.5 
14.5 


3.64 
30.4 
13.0 


San pi a S1zo° 


297 


147 


444 



SOURCE: Celculetlone from HDRC Client Informetlon Sheets. 

NOTES: DIs- Ibutione may not edd exectly to ? 00 , 0 percent beceuse of 
ro un d1 n g. 



All members of the semple ere femele single h, eds-of-househol d. 

( con t1 n ued ) 
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TABLE 2.3 [continued ) 



Differences between research groups ere stetisticelly significent 
using e two-teiled t-test or chi-squere tost et the following levels: • = 10 
percent? = 5 percent; = 1 percent. 

e 

Distributions raey not edd to 100.0 percent because semple members 
cen heve children in more then one category. 

b 

Due to missing date, the semple sizes for average number of 
months since most recent WEET registration ere es follows: 279 for 
experi mentals, 137 for controls and 416 for the total. 

c 

Celculeted frna Unemployment Insurance eernings records from the 
State of Maine. These deta include zero values for semple members not employed. 

d 

For questions concerning longest job, semple sizes ere besed on 
the number of people who report e longest job on the Client Informetion Sheet. 
Due to missing date, for selected ch e rec t e r i 8 t i c s , these semple sizes very es 
follows: 125-127 for e xpe r i me n t e I s , 65 for controls end 190-192 for the totel. 

e _ 

For selected characteristics, semple sizes mey very up to 19 
semple points due to missing data. 



frequenci es. 



Chi-squere test ineppropriete due to low expected cell 
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significant differences included: age, race, age of youngest child and 
prior employment history, controls were slightly older than experimentals 
(31,1 vs. 2 9.6 years); more controls than experimentals were white (100 vs. 
96.9 percent); and fewer controls than experimentals had children under six 
years old (42 vs. 53 percent). The average earnings in the year prior to 
randan assignment (including values of zero for those who did not work) 
were higher for controls than experimentals ($683 vs. $448); the average 
number of months employed during the two years prior to random assignment 
was also larger for controls than for experimentals (3.8 vs. 2.8 months). 
The impact analysis includes adjustments for these differences. 

Compared to a profile of the AFDC caseload in Maine in January 1983, 
the TOPS research sample showed basic similarities regarding ethnic mix and 
length of welfare receipt. 5 The two groups were alike in being predomi- 
nantly female single heads-of-household and white — reflecting the ethnic 
composition of the state. They were also alike in the duration of their 
most recent stay on AFDC: for approximately one-third of both groups, more 
than two years had elapsed since their most recent case opening. 6 Compared 
to the WEET caseload, however, the TOPS sample showed some important 
differences. For example, a smalxer proportion of sample members than WEET 
registrants had completed less than 12 grades of school (33 percent vs. 42 
percent). Similarly, a smaller proportion of the TOPS sample had a ninth 
grade education or less (15 percent vs. 27 percent). 7 The TOPS research 
sample also included a higher proportion of women exempt from WIN 
registration requirements. 8 

These comparisons suggest that the screening process for TOPS yielded 
a sample that is not typical of the WEET and AFDC caseloads on some key 
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measures. Overall, a number of factors — the high percentage of sample 
members who are divorced or widened, the presence of pre-school children, 
the relatively high level of educational attainment and the absence of 
employment history — suggest that the wemen in the TOPS sample may 
represent a displaced hememaker population, that is, wemen who either have 
no previous employment or who discontinued working for marriage and family 
and then fell on hard times. 

The TOPS sample is also quite different fran the research samples in 
other state initiatives evaluated by MDRC, which excluded individuals not 
subject to the mandatory registration requirements in WIK. Maine had the 
highest percentage of sample members with high school diplomas or GEDs (75 
percent) and the highest proportion of recipients who had their own AFDC 
case for more than two years (63 percent). The TOPS sample also had a high 
proportion of members with children under the age of six (49 percent); only 
Arkansas, which required their participation, had a larger proportion of 
pp*^ts with children in this age group (54 percent). (See Appendix A-) 
The*e comparisons with other demonstration states further illustrate that 
the TOPS sample members were a group of screened individuals who, although 
they had fairly long-term welfare dependency and little prior labor force 
attachment, were well educated and showed interest in participati , 

II. Data Sources 

Numerous data sources were used in this evaluation. Administrative 
records were used to measure outcomes and participation rates. Other 
documents were used to tally demographic characteristics and more detailed 
participation information. Much qualitative information, mainly interview? 



with program staff, was used m conjunction with this quantitative 
information. Each data source is described below. 
A * Client Information Sheets 

The Client Information Sheet { CIS) , designed by MDRC and filled out by 
program staff at the point of randun assignment, was the major source of 
demographic and socioeconomic characteristics of each sample member. It 
provided information which helped document the char-rteristics of the TOES 
sample. The CIS included data on age, sex, ethnicity, family composition 
and educational attainment as well as basic information on welfare and 
employment histories, with particular attention given to each sample 
member's experiences during the two years prior to random assignment. The 
CIS data were complete for 95 percent of all sample members. 9 
B. Administrative Records 

Administrative records were the primary data source for the impact and 
,enef it-cost analyses." Table 2.4 summarizes the types of records data 
used, and the length of follow-up for each quarter of randan ,«signm e nt. 
1 * The state of Maine U nemployment Insurance System 
The Unemployment Insurance (UI) system provided data on the 
earnings of the sample members, by calendar quarters." These individual 
earnings amounts include earnings received iron the OJT employers as well 
as unsubsidized employers. Using social security numbers to identify the 
sample members, MDRC collected data from the automated earnings file for 
each sample member frccn one year prior to random assignment until the end 
of the second quarter of 1987. The first calendar quarter, the quarter of 
random assignment, can include earnings before random assignment. 
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TABLE 2.4 



HAIhc 



UNGTH OF AVAILABLE FOLLOW-UP BY DATA SOURCE AND PERIOD 0? RANDOM ASSIGNMENT 



I 

I 







I 

Point at Which 
Data Collection 
Begine 




Length of Follow-Up By Period of Random Assignment 


Oeta Source 


Lest Dete Oats 
Are Aval Labia 


October - 
December 1983 


January - 
March 1884 


April - 
June 1884 


July - 

September 1884 


October - 
December 1984 


WEET Tracking Record** 


November 19B5 


Date of 
Random Assignment 


Twenty-Three 
Months 


Twenty 
Months 


Sever ..een 
Months 


Fourteen 
Months 


Eleven 

a 

Monthe 


JTPA Tracking Records 


Novesbar 1885 


Date of 
Random Assignment 


Twenty-Three 
Months 


Twenty 
Months 


Seventeen 
Monthe 


Fourteen 
Monthe 


Eleven 

8 

Months 


Querterly Employment 
and Earnings 


Sacond Calendar 
Quarter of 1887 


Four Quarters 
Prior to 
Psndom Assignment 


Fourteen Qujrtere 

After 
Random A&elgnment 


mil boon uusriDrb 

After 
Random Assignment 


Twslve Quart *& 

After 
Random Assignment 


Eleven Quartet e 
Aftar 

Randftt Atolgnraent 


Tsn Quarters 
Aftar 
Random Assignment 


Monthly AFOC Grant 
Payments 


Novbaber 1987 


February 1883 


Forty-Eight 
Months 


Forty-Five 
Monthe 


Forty-Two 
Months 


Thirty-Wine 
Monthe 


Thirty-Six 
Months 


Weekly Unemployment 
Znsursnce Benefits 


December 1888 


Tmelva Months 
Prior to 
Random Assignment 


Thirty-Seven 
Months 


Thirty-Four 
Months 

i 


Thirty-One 
Month* 


Twenty-Eight 
Months 


Twenty-Five 
Monthe 


MrtTr . e„ 







b 

Employment and earnings data ara uased on Unemployment Ineurance eorninge racorda, which report earning on a calender quarter beele Sinra 
L!_! q08r ? frend< * a " 8t fl nwnt » contain aoae earnings fro. t: • period prior to rondo- aaeignaent. it la excluded fro. the follow-up period 



quarter 1 

for employment end earninge. 



c 

, . , ." r8t *° nth ° roUo "~ up for " 8lfero orent peymente la the month in nhich an individual >ee rendo.ly eeeignad. For the i.pect enalyeie, 
monthly date for welfare grant p.yenta ere eogregeted into calender quartere to «atch eaployment and earnings date. Since quarter 1. the quarter of 
rendom eoe.gnment, ia excluded fr^ the follo»-up period for employ.ent ond eerninga, it (e also excluded fro. the folloa-up pariod for ^esures of AFOC 
race i p t » 
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Therefore, it is not counted as a follow-up quarter in the impact analysis. 

Weekly Unemployment Insurance ccmpensation amounts paid to the sample 
m*nbers were also collected. Records of UI benefits were obtained through 
a manual process of matching the social security numbers of the sample 
members with a list of those individuals receiving any compensation from 
one year prlor to randCffi asslgnment ^rou^i December 1986. These UI 
benefit amounts were used in estimations for the benefit-cost analysis. 

It is important to understand that, in sane instances, the UI data 
system underestimates income because of unreported earnings. The UI 
measure of earnings does not include of f-the-books earnings, or the 
earnings of people who have moved or work out of the stats. Also, 
employers may not report the earnings. 

Because of the possibility of under- reported earnings, measures were 
taken co determine the accuracy and completeness of the Maine UI earnings 
data. A comparison of self-reported employment prior to random assignment 
from the CIS and UI earnings records revealed a slightly larger, though 
still small, proportion of discrepancy than is found from the same 
comparison ir. ether state initiatives MDRC has evaluated." This does not 
constitute a major limitation for the evaluation, however, because thore is 
no reason to believe there were differences in the reporting of experi- 
mental and control earnings, and therefore no reason to suspect bias in the 
experimental-control differences. 

2- The State of Maine AFDC payments system 

This data set supplied the actual monthly AFDC payment amounts 
received by each recipient. The state's computerized system, which issues 
all automated AFDC grant checks, was the direct source of this information. 
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These monthly payments wei aggregated into calendar quarter amounts to 
match earnings. 13 Therefore, the calendar quarter cf ran**, assignment is 
not considered a follow-up quarter in the impact analysis for the AFDC 
data. 

For all sample members participating in TOPS OJT and therefore having 
a portion of their grant diverted, these AFDC grant amounts represent the 
residual payment amount (i.e., the amount paid directly to the client). 
The diverted grant amount (i.e., the amount put in the grant diversion 
pool) is not included in reported AFDC outcomes for individuals, if an OJT 
participant had her entire grant diverted (i.e., the amount of her earnings 
made her ineligible to receive an AFDC grant payment) she would remain in 
OJT but have no AFDC payment. Therefore she would not appear as an AFDC 
recipient in the outcome measure of AFDC receipt in the impact analysis. 

AFDC data were collected from February 1983 through November 1987, 
For each sample member, therefore, there are data beginning at least eight 
months prior to random assignment and for 36 subsequent months including 
the month cf random assignment. 1 * 

3 . WEET Information System 

The WEET Information System provided computerized data on program 
participation in services provided by WEET as well as information on job 
placement and deregistration. Information was collected from October 1983, 
the beginning of random assignment, through November 1985. Sample members 
have between 11 and 26 months of participation follow-up data, depending on 
their date of random assignment. Additional participation information from 
the manual tracking logs kept by caseworkers was incorporated into the WEET 
data to supplement knowledge of program activity within TOPS and WEET. 



4 . JTPA Information System 

The JTPA system provide*] information on participation in all of 
its services. Thee data were used to monitor additional services received 
outside of TOPS. Of primary interest was its use in measuring the level of 
services received by the control group. These data were collected for the 
same period as the WEET tracJr \ng data, from October 1983 to November 1985. 
C On-The-Job Training Contracts 

Copies of the OJT contracts written by WEET, or by JTPA for WEET, were 
collected for all experimentals placed in grant-diversion funded OJT posi- 
tions through TOPS from October 1983 through December 1986. The contracts 
written for each OJT placement supplied sane necessary supplemental informa- 
tion about on-the-job training. They were used to calculate average wages, 
hours and scheduled duration, m conjunction with supplemental information 
gathered from the WEET staff, the actual number of weeks of participation 
were also calculated. in addition, the contracts supplied descriptive 
information about the occupational distribution of these placements. 

D. Worksite Survey 

MDRC staff conducted telephone and worksite interviews with a small 
sample of supervisors of clients in work experience and OJT positions. 
This information was used for descriptive purposes and to estimate seme 
components of the benefit/cost analysis. 

E. Field Research 

MDRC staff made repeated trips to state and local WEET and JTPA 
offices to interview staff about administrative arrangements in Maine, the 
background of welfare reform in the state, the JTPA and WEET systems, and 
local variations in TOPS services and structure. 
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CHAPTER III 
PARTICIPATION IN TOPS 

This chapter discusses the institutional setting of the TOPS demon- 
stration, the movement of the experimental group through the sequence of 
TOPS activities, and the characteristics of the OJT positions* It also 
describes how each of the TOPS components was implemented, paying 
particular attention to the job development process in the OJT phase* 

I. Institutional Setting 

The TOPS sequence? was designed to prepare women for jobs by providing 
opportunities to enhance their self-esteem, develop mature work attitudes 
and acquire experience and training in a supervised work setting. The 
combination of services that constitute the program model add up to an 
intensive program treatment, as indicated by a number of different 
measures. First, the sequence was lengthy: it typically took participants 
in all three components 11 months to complete the full cycle. With a few 
exceptions, enrollees first participated in pre- vocational training, last- 
ing two to five weeks? they were then assigned to work experience positions 
lasting for up to 12 weeks; they were then eligible for OJT positions for 
up to six months. Second, both the unpaid work experience and OJT 
components, as implemented, provided substantive experience in real vork 
settings. Third, during the pre-vocational training and work experience 
components, TOPS participants were the recipients of considerable amounts 
of staff time and attention. Fourth, the program was relatively expensive 



to run,, costing an average of $2,627 per experimental. About half of this 
figure represents the cost of support s-ervices and allowances, case manage- 
ment, interagency coordination and related administrative activities; this 
underscores the emphasis placed on support services and case management as 
critical elements of employability c"-»velopnent in the TOPS program. 

Both WEET and JTPA staff indicated that their work with TOPS partici- 
pants required a greater investment of time and energy than did the ser- 
vices provided to other WEET or JTPA enrollees. This reflected the smaller 
than usual staff-to-client ratios in the TOPS pre-vocational classes, and 
the addition of certain activities, i.e., support groups and job search, 
not envisioned in the design, in addition, WEET staff estimated that they 
spent extra time counseling participants, arranging support services, 
developing work experience placements, monitoring clients' progress through 
the lengthy sequence and coordinating activities with staff in other 
agencies. All these extra functions are reflected in the costs of the 
program which are generally higher than comparable activities offered in 
other welfare employment programs MDRC has evaluated since 1981. 

All WEET staff worked with TOPS participants in addition to their 
usual caseload. The orJLy staff responsible just for TOPS were five full- 
time .'coordinators,; one per region, hired specifically for the demonstra- 
tion project. They were initially employed to maintain the reporting data 
necessary for the research, but sane of the coordinators eventually took on 
a more substantive role in recruiting, monitoring and serving as liai s 
with the JTPA staff. JTPA staff who worked on TOPS also maintained other 
responsibilities for JTPA programs and were not assigned just to TOPS. 

The progress of the TOPS experimental was closely monitored by WEET 
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staff who, in addition to keeping WEET 1 s usual case file records, filled 
out monthly ^participation logs; indicating the completion of each week of 
activity and any reasons for interruption or termination. 

The implementation challenges in mounting the demonstration were 
considerable, as discussed in the interim TOPS report. 1 Both WEET and JTPA 
staff were still adjusting to their new roles when the demonstration began. 
Moreover, the thrust of TOPS did not always fit neatly into the WEET 
structure. There was tension, for example, between the stx * central 
office role needed to ensure the demonstration was implemented as planned, 
and the encouragement of local diversity and line staff flexibility that 
was a hallmark of the new WEET system. In particular, staff in sane local 
offices questioned whether OJT training was more appropriate than classroom 
training for all those who met the TOPS eligibility criteria. As a result, 
they tended to be more selective in recruiting participants than central 
staff had anticipated. They also felt that the- amount of time involved in 
operating TOPS interfered with their ability to serve their full caseload 
of clients. 

Other difficulties were encountered in carrying out the interagency 
coordination. In theory, the two systems had good reason to collaborate. 
WEET staff could draw upon JTPA's greater experience in job development and 
placemen as well as its financial resources. JT£A agencies gained both a 
referral source that could help them fulfill their requirement to serve a 
quota of AFDC recipients and a funding source to provide support services 
that were otherwise unavailable under the JTPA legislation. 

In practice, hewever, coordination presented numerous difficul 
WEET and JTPA staff had quite different philosophies about working with the 
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disadvantaged — staging in part from the different measures by which 
their performance was judged - and different standards for determining 
whether participants were job-ready. JTPA staff tended to be more 
stringent and to value more heavily the needs of the employer; WEET staff 
were more protective of clients and less focused on ths employers' perspec- 
tive. Since JTPA staff were largely responsible for the OJT development, 
the JTPA philosophy had important repercussions at this critical stage in 
the TOPS sequence. 

Initially, poor communication channels, the absence of a central 
authority to decide between competing priorities, and different expect- 
ations about staff and client roles m each phase of TOPS made the 
transition between components difficult. They also complicated the task of 
monitoring the progress of participants through the entire sequence. 
Working out these differences required considerable adjustments in the 
roles and responsibilities of each local office as the demonstration 
progressed, as well as frequent refinements in the content of the services 
offered in the TOPS components, staff in both systems worked hard to irot. 
out their differences, and coordination ir. roved over time. Nevertheless, 
tension between the two agencies wade the implementation of TOPS more 
difficult, particularly during the early stages of the demonstration. 

II- Participant Flow in tops 

As would be expected with a program for which clients applied and were 
then screened for appropriateness, participation in the first TOPS activity 
was relatively high. a total of 89 percent of the experimental sample 
participated in a TOPS pre-vocational class within the unif 01. , follow-up 
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period ; 12 months after randan assignment (Table 3.1). Participation 
rates in the other components were progressively lower: 68 percent of the 
experimental were placed in a work experience slot within 12 months and 29 
percent were placed in an OJT position within 12 months, A detailed study 
of participant flew among the sample randomly assigned b tween October 1983 
and March 1984 indicated that the fall-off occurred primarily between 
components: if participants entered pre-vocational training or work 
experience, they tended to complete the activity. 2 

No additional experimental entered pre-vocational training or work 
experience after the 12-raonth follow-up period, but six TOPS enrollees 
entered OJT more than a year after random assignment. Although the 
research design specified that TOPS enrollees should not participate in 
training or long-term education, a substantial minority of the experimental 
sample did enroll in such services. (See Table 3.2.) Those enrolled in 
education reflect both those who participated in basic education — as 
allowed in the TOPS design — and those who enrolled in more advanced or 
longer *;erm courses. In addition, about one-third of tuie experimental 
participated in group job search, Individual job search, or both. This 
reflects the fact that, as discussed later in this chapter, the parti- 
cipants* cwn job search became an important aspect of the OJT development 
phase in TOPS. 

During the entire follw-up period, the largest proportion (?Q 
percent) of enrollees participated in a cctfiLination of pre-vocational 
training and work experience; 27 percent participated in all three TOPS 
components; 19 percent participated only in pre-vocational training. (See 
Table 3.3.) A number of factors account for the declining level of 



TABLE 3.1 



MAINE 

TWELVE-MONTH PERFORMANCE INOICATORS 
FOR EXPERIMENTALS 



Pe r f o rmence 


Ind 1 ce t o r 


Nuabe r 


Pe rcen t 


Per t1 d pated 
Componen t 


in Any TOPS 


266 


89.6 


Parti ci pe ted 
Training 


1n Pre-Vocetionel 


263 


88.6 


Per tl c 1 1 at ad 


i n Work Exper :ance 


201 


6/. 7 


Parti c1 pated 


in On-the-job Treinlng 


85 


28.6 


Entered Employment 8 


57 


19.2 


Semple Size 


297 



SOURCE: MORC celculations from WEET Informetion System end JTPA 
Informetion System dete. 



NOTES: Semple members rendomly essigned in Oacember 1984 heve between 11 
end 12 months of pe r t i c i pe t i on follow-up Jete, depending on whether they were 
rendomly essigned in the eerly or Later part of Oacember. These semple members 
ere considered to neve 12 months of pe r t i c i pe t i on follow-up dete. 



one day. 



Perticipetion Is defined e attending an activity for et leest 



Job plesement Informetion Is besed on employment thet is 
reported to WEET steff, This percentege represents semple members who entered 
employment and did not pertldpete 1n On-the-job treinlng. Progrem plecement 
dete will not be used to meesure impects. 
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TABLE 3.2 



MAINE 

PARTICIPATION IN WEET A ,0 JTPA SERVICES FOR EXPERIMENTALS 
OVER THE ENTIRE FOLLOW-UP PERIOO 



Pe rf o rmence 


Xnd i ca t o r 


Numb e r 


Percent 


Parti cipeted 


in Any Activity 


276 


S2.9 


Participated in Either Pre-Voce t i on a I 
Treining, Work Experience, or On-the-Job 
Tre i n i ng 


266 


69.6 


Parti cipeted in 

Pr e-Voce t i one L Treining 


263 


88.6 


Par ti c i pet ed 


in Work Experience 


201 


67.7 


Par- ti cipeted 


in On-the-Job Treining 


91 


30.6 


Perti cipeted 


in Group Job Seerch 


26 


8.b 


Pa rti cipeted 


in Individual Job Seerch 


92 


31 .0 


Participated 


in Educetion 


18 


6.1 


Pa rti cipeted 


in Treining 


9 


3.0 


SempLa Size 


297 



SOURCE: MORC celculetions from WEET Informetion System end JTPA 
Informetlon System dete. 

NOTES : SempLe members heve between 11 end 26 months of 
participation follow-up dete, depending on their dete of rendom assignment. 
The*; is, eempLe members rendonly essigned eerly in the demonstreti on heve 
more follow-up then sempLe members rendonly essigned Leter in the 
demonstreti on. 

Per ti ci pe ti on is denned es ettending en ectivity for et 

Le e st one d ey . 
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TABLE 3.3 



MAINE 

PARTICIPATION PATTERNS IN TOPS ACTIVITIES FOR EXPERIMENTALS 
OVER THE ENTIRE FOLLOW-UP PERIOO 



Performance Indicator 


Numbe r 


Pe rcen t 


Pertlcipjted in Pre— Vocationol Tmfn<inn 
On ly 


57 


19.2 


Perticipeted in Work Experience Only 


A 

1 


0.3 


Perticipeted in On-the-job Treining 
Only 


0 


0.0 


Perticipeted in Pr e-Voce ti on e I TreJ.iing 
end Wck Experience Only 


11? 


39.4 


Perticipeted in Pr e-Voce t i o ne I Treining 
end On-the-job Treining Only 


8 


P 7 


Perticipeted in Work Experience end 
On-tha-Job Treining Only 


2 


0.7 


Perticipeted in the TOPS Sequence 
[Pre-Voce tionel Treining, Work 
Experience, end On-the-job Treining] 


81 


27.3 


Oid Not Perticipete in Any TOPS 
Ac t i v i ty 


31 


10.4 


Sample Size 


297 


100.0 



SOURCE : MORC cc i -s u La t i or,a from WEET Informetion System end JTPA Informeti 
System date. 



NOTES: SempLe oeobare heve between 11 end 26 months of pe r t i c i pe t i on 

follow-up dete, depending on their detc of rendom assignment. Thet is, semple 

members rendomly essigned eerly in the demonetretion heve more follow-jp than 
semple members rendomly essignod Leter in the demon s t re t i on . 

Perticipetion is defined es ettending en ectivity for et leest one 

dey. 
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participation in the subsequent components in the TOPS sequence. sane 
people got jobs on their own, went off welfare for other reasons* or moved 
out oZ the area. Seme ceased to participate because of problems with 
health, child-care arrangements or other family situations. Others lost 
interest in the program or decided that they wanted sane other type of 
training or education. 3 Still othu: sample members were considered 
inappropriate for services at later screening points in the program, as 
discussed below. 

Only two members of the experimented group were sanctioned for non- 
participation. This reflects both the large proportion who were exempt 
from mandatory participation requirements and the general philosophy of the 
WEET program. Staff wanted to work with AFDC recipients who were interest- 
ed in participating; there was no intent to sanction in order to reduce 
welfare costs. If clients were sanctioned, it was because of other 
problems in addition to non-ccmpiiance. 

Most experimental entered the pre-vocational training and work 
experience components of TOPS within a few months after randan assignment, 
but participation in OJT rse more slowly and continued to climb throughout 
the 12 months following randan assignment. (See Figure 3.1.) On average, 
participants in OJT began working in the seventh month after randan assign- 
ment. Given the fact that the maximum duration of the pre-vocational and 
work experience components was five weeks and 12 weeks, r6 actively, this 
suggests that there was a significant lag between the end of one component 
and the start of the next. Participants in all three components took 11 
months to complete the iull TOPS sequence. 4 All but a few participants 
completed their OJT assignments within 18 months after randan assignment. 



PGURE 3.1 
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TRENDS IN CUMULATIVE PARTICIPATION RATES FC r 
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Month of Follow-Up 

SOURCE AND NOTES: See Table 3.1. ; g 7 
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For this reason the six quarters after randan assignment are considered the 
in-prograra period and follow-up quarters 7 through 11 are considered the 
post-program period for the impact analysis in Chapter V, 

Participation rates by subgroup and region are not presented here 
because the sample sizes are too small to make such comparisons meaningful, 

III* Implementation of the TOPS Components 

A « Implementation of Pre-vocational Training 

The pre- vocational training wcs designed to prepare participants for 
work by familiarizing them *tth job opportunities, employer expectations 
and job search techniques. The content of the ti .ining diff e-ed by site 
and by cycle as staff updated the materials to better meet enrollees 1 
needs, but all materials were modifications of two basic curricula, cv>e, 
developed for JTPA enrollees, emphasized employer expectations, appropriate 
behavior on the job, and how to obtain and keep & job. The other, 
developed by the Maine Displaced Houemakers Association, placed more 
emphasis on helping women to make career choices and decide what they 
wanted to get out of a job. Both curricula included modules on goal 
setting, values clarification and decision-making, and household budgeting, 
as well as job hunting, resume writing, and interviewing skills. staff 
also discussed how going to work would affect participants' A?DC grants, 
and taught them how to be their own advocates within the AFDC system. 
Particular attention was paid to using the pre-vocational classes to 
increa. participants' self-esteem and their expectations about what tluy 
could accomplish by going to work. The format included a mixture of 
lec±ures and group discussions; enrollees were expected to participate 
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actively and complete a number of exercises in class and at hone. Group 
dynamics were considered a key part of the approach. 

The length of the course varied from site to site, but generally was 
between 12 and 20 days, spread over a four- or five-week period. Most 
sites scheduled pre-vocaticnal classes in both the morning and afternoon; 
in a few locations, time was set aside for remedial education or individual 
counseling. The pre-vocational classes, for the most part, were quite 
small, ranging from four to 21 participants; 25 out of the 30 classes had 
ten or fewer persons enrolled, with an average class size of seven. With 
the exception of two cycles, the TOPS participants were In pre-vocational 
classes by themselves, rather than mainstreamed with other WEET or JTPA 
enrollees. staff opinions on the optimal size for a pre-vocational class 
differed, although all agreed that relatively small classes were best. In 
two WEET offices, staff felt that cycles ran best with six to nine 
participants, due to the amount of staff time required per participant; in 
another location, WEET and JTPA staff felt that ten to 12 ..as a better size 
to achieve the desired group dynamic. 

Staff were enthusiastic about the pre-vocational training in TOPS and 
reraarkea on the noticeable improvements in the attitudes of many particle 
pants. However, they agreed that pre-vocational training alone was not 
sufficient job preparation for the TOPS enrollees. After a few partici- 
pants who were placed directly in OJT after pre-vocational training 
encountered sane problems on the job, it became accepted policy for all 
participants to enroll in work experience prior to an OJT placement. 
B « implementa tion of the Work Experience Compone nt 
The work experience component in TOPS was intended to develop mature 
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work attitudes in a real work envi-onment; staff felt it had less potential 
for substantive skill development. In practice, work experience also 
served as a way for staff to evaluate participants 1 job readiness and, 
thus/ their appropriateness for an OJT placement. For sane participants, 
the unpaid work experience position developed into an OJT position. As 
reported in the interim study, through Septenber 1984, 12 women had been 
hired as OJT employees by their work experience supervisor out of a total 
of 41 women who had participated in both work experience and OJT, 5 

Over 6 0 percent of the work assignments were in clerical positions, 
and most of the remainder were in other traditionally 'temale' areas of 
work. (See Table 3.4.) The standard length of the work experience 
positions varied between eight and 12 weeks, depending upon the cycle and 
the location, and the program model was limited to 20 hours of work per 
week. (This 12-week maximum could be waived on an individual basis.) WEET 
staff were responsible for developing and monitoring the work experience 
positions in all but three locations. Where WEET was responsible for 
implementing the component, the work experience positions had to be in the 
public or non-profit sector. Elsewhere, placements with private employers 
accounted for only a small proportion of all « ^rk experience positions in 
an e*rly sample. 6 Nevertheless, some staff felt the restriction on private 
sector placements severely limited the potentiax icx turning work experi- 
ence positions into OJT slots, since the OJT phase was targeted to private, 
for-profit employers. 

Most offices tried to individualize the slots to meet participants' 
job interests. This lengthened the development process, and accounts for 
much of the lag time between the completion of pre-vocational training and 
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TABLE 3.4 



MAINE 



DISTRIBUTION Of WORk EXPERIENCE PARTICIPANTS 
BY JOB CLASSIFICATION 



Job Classification 


Nurabe r 


Pe rce n t 


Clerical Occupations 


119 


62.3 


Service Occupations 






Soci al Se rvi ces 


30 


15.7 


Food Service 


9 


4.7 


Health Cera 


8 


4.2 


Seles/Casnier 


6 


3.1 


Manual Skills Occupations 


13 


6.8 


a 

Other 


G 


3.1 


b 

Sample Size 


191 


100.0 



SOURCE: MORC calculations from WEET Central Offico records of Work 
Experience job titles. 

NOTES: Distributions may not add exactly to 100.0 percent beceuse of 
roundi ng . 

Job clesci fi cations are besed on the U.S. Oapartment of Labor's 
occupational titles from the Dictionary o f_0ccu £ a t i one I Titles, fourih edition, 
1977. 

Participation is defined as attending Work Experience for at least 

one day. 

a 

Includes generel labor assistant, g round s ke e pe r , horticultural 
aide, meintenance worker, and Laundry workers. 

b 

A total of 201 experimental participated in Work Experience. Ten 
sxperi men-.als were excluded from the above celculations due to missing job 
classi fication da'.e. 



the start of work experience, sane sites arranged it so that all the work 
experience positions began immediacely after the end of pre-vocational 
training; in others, entry was staggered over sane weeks* 

In many locations, staff began to run support group sessions for the 
participants in work experience jobs. These meetings had two purposes: to 
help participants deal with problems on the job and the pressures of 
juggling family and work commitments, and to begin planning for the time 
when they would enter paid employment. In seme locations, the sessions 
also included work on resumes *nd job-search skills^ The frequency and 
length of the support groups varied frcm site to si' and cycle to cycle, 
but sessions were generally scheduled weekly or bi-weekly for one to two 
hours. Staff indicated that attendance was frequently sporadic, because 
enrollees had many commitments competing for their time. 

Monitoring practices differed frcm site to site* Sane required work 
experience supervisors to fill out weekly evaluations; others asked for 
monthly reports. Many relied on verbal canments rather than written forms. 
WEET staff in some locations also made worksite v .si< 3. During this 
component, participants received an allowanc< of $1.25 an hour to cover 
training-related expenses, in addition to their AFDC grant. Most offices 
required the women to come to the WEET office weekly to pick up the 
allowance check in order -to give stafi a opportunity to talk with the 
participants about their experiences. 

MDRC interviews with the supervisors of 17 TOPS participants placed in 
work experience between May and November 1984 suggest that the work experi- 
ence component fulfilled the intended purpose of introducing the women to a 
work setting aid giving them the opportunity to learn and practice good 
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work habits. 7 The interviews indicated that the positions were generally 
entry level and would have paid between $3.00 and $5.30 an hour. The 
majority of supervisors found the participants to be at least as productive 
as regular new employees, if not more so. Most participants were judged to 
be the same or better than the average new worker in a number of basic work 
habits, such as punctuality and attendance. Overall, TOPS participants 
also were judged to have the requisite cognitive skills and general work 
skills at the start of their assignment. However, among those in positions 
which required specific occupational skills, two-thirds were considered 
inadequate when they started the job. In addition, over half of those In 
jobs requiring them to work without supervision were considered inadequate 
in this skill when they began. Between one- third and one-half of the 
participants were judged to have improved their performance in these areas 
by the time of the interview. 

B « Implementation of the OJT Component 

That participation in the OJT component was lower than in the preced- 
ing activities in the TOPS component is not unexpected. First, in a 
sequenced program like TOPS, enrollees who stopped participating at an 
earlier point in the program would not be eligible for placement at the OJT 
stage. Thus, the pool of potential placements was narrowed from the 2 97 
experimental to the 201 individuals who entered work experience. (Another 
eight individuals entered OJT directly after pre-vocational training.) 
Second, some participants who completed pre-vocational training and work 
experience found unsubsidized employment or went off AFDC for other 
reasons, and did not look for an OJT placement. For the 118 participants 
who took part in work experience but did not participate in OJT within the 
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month after they should have completed work experience, MDRC calculations 
show that 33 had unsubsidized jobs and 5 others were no longer on AFDC. 8 
From this perspective, the placement rate into CUT rises from 31 percent of 
the whole sample to 56 percent of *^ose who entered >ork experience, were 
still on AFDC, and did not have an unsub dized job within che month after 
their work experience was scheduled to end. 

These factors are not sufficient, however, to explain why the CUT 
assignment rate in TOPS was so much lower than participation in the other 
TOPS components. Employer reluctance to hire AFDC recipients or to use a 
subsidy is a possible factor, although the fact that a large number of AFDC 
recipients were placed in CUT positions by both WEET and JTPA during the 
demonstration period suggests that TOPS did not exhaust the potential 
market for subsidized workers. 9 This point is reinforced by the fact that 
CUT positions for TOPS participants were marketed no differently than these 
other placements. More important as determin^-ts of the number of TOPS 
placements, as described below, were two oth* aspects of the TOPS CUT 
development: the job development process « 1 ir luded additional 
screening to determine a TOPS participant's app„ lateness for an CUT 
placement, and the heavy reliance on the clients' owi. isarch efforts. 
1. The Job Development Process 

Except in a few locations, the primary responsibility for placing 
TOPS participants in CUT positions lay with JTPA staff. In most locations, 
job development was done by specialized job developers who had placement 
responsibility — subsidized and unsubsidized — for all JTPA enrollees, 
and who did not work just with TOPS clients. Although WEET staff made 
efforts to have the job developers meet the TOPS participants earlier in 



tha program, mcsc job developers had little or no contac with the 
prospective employees before the start of the OJT development phase. 

In general, JTPA staff made few efforts to individualize CJT positions 
for TOPS clients, in the sense of developing a job specifically for a 
particular client. instead, the primary sources of OJT positions for the 
TOPS clients were referrals ' M the pool of employment opportunity 
previously identified by job development staff, jobs identified by clients 
through their on. job search activities, and rollovers from the work 
experience assignments. As discussed above, 12 out of 41 OJT contracts 
written through September 1984 were developed out of work experience 
positions. 

The staff role in the OJT development phase included: supervising the 
job search efforts; providing leads, encouragement and advice; and working 
directly with employers to develop a pool of available positions for JTPA 
clients (including but not limited to TOPS). They also made referrale for 
promising candidates to employers who were hiring. 

Once TOPS participants were referred to JTPA for job placement after 
completing their work experience assignments, it was made clear to them 
that they were expected to be active in their own job development — to 
answer employment advertisements in the newspapers, send out resumes, and 
call prospective employers identified through the Yellow Pages or through 
staff contacts. m short, they were expected to apply the job search 
strategies taught in the p: . vocational component, over We course of the 
demonstration, the trend was for staff to intensify an* formalize the job 
search aspects of the OJT development phase. Initially, staff in most 
locations worked with TOPS clients on a one-to-one basis; some job 
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developers monitored progress by havincj TOPS participants renr^t weekly or 
bi-weekly? others left it up to the enrollees to initiate contact. in a 
few sites, staff chose to work with the TOPS women as a group. *>y the 
final cycles in a few locations, TOPS participants were placed in formal 
job clubs « sometimes with other JTr. or WEET clients, sometimes by 
themselves. This involved intensive, sup rvised job search activity in the 
office, for a week or more at a time. 

TOPS participants who located a job on their own were still eligible 
for the OJT subsidy. JTPA job developers — and sometimes WEET staff ~ 
generally informed the employer c.bout the conditions of the OJT subsidy 
and, on occasion, were instrumental in negotiating a higher wage level than 
the employer initially proposed. JTPA offices varied in their approach 
to marketing the OJT subsidy. sane instructed their clients to explain 
that they were candidates for OJT when they applied for a job; others 
advised clients u^t to raise the issue until the employer showed interest 
in hiring them. So^e staff wanted OJT candidates to explain how the 
subsidy worked; others wanted prospective employers to call them directly, 
and discouraged 1ob applicants from providing anything but th* most basic 
information. 

In order to be considered for a referral to an OJT position in JTPA 1 s 

previously developed pool of jobs, the TOPS participants had to go through 

another, often informal, screening at the end of their work experience 

assignment. Interviews with job developers across the state indicated that 

they were unlikely to refer candidates to OJT positions if: 

* candidates were still unsure about the type of job they 
wanted; 
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• their skills were inadequate or inappropriate for the type of 
job they were seeking; 

• they lacked ^work maturity,'; 

• they were not really interested in working. This category 
included those who were worried about losing their Medicaid 
coverage, housing subsidy, or AFDC eligibility if they went to 
work. 

Job developers got a sense of where TOPS participants stood on these 
issues by interviewing them, reviewing evaluations of their performance in 
the pre-vocational training and work experience components, and observing 
their conduct in carrying out their own job search. TOPS participants who 
did not contact the job developer on a regular basis, failed to follow 
through on job leads provided by the job developer or did not execute other 
assignments as directed, demonstrated that they were not job ready; job 
developers were unlikely to make job referrals or aggressively pursue 
potential employers for these candidates. Therefore, how aggressive the 
job search became depended largely on the clients' own efforts. 

JTPA job development practices caused some friction with the WEET 
staff who had expected that job developers would be more aggressive in 
finding ojt positions and would tailor them to the interests of individual 
clients in TOPS, as WEET staff had done in developing the work experience 
positions. over time, however, WEET staff also became advocates of the 
client-initiated job search. in one office, for example, WEET took the 
initiative and placed TOPS participants for whom JTPA had not found an OJT 
into a three-week mini-job club run by their own staff. Another WEET 
office, which split responsibility for OJT placements with JTPA, also 
placed TOPS participants in a formal job club by the final cycle and used 
less structured job search activities in previous cycles. One important 



difference remained: WEET staff tended to place greater emphasis on the 
need for structured, formal job search in a group setting, rather than 
relying on staff- supervised independent job search for the TOPS clients. 

The reliance on client-initiated job development in TOPS reflected 
both resource constraints and the conviction of JTPA staff that client job 
search is a highly effective placement strategy. However, it has been 
generally assumed that in an OJT program, because less job-ready applicants 
will require extra help in order to be hired, staff 'marketing* of poten- 
tial employees — as well as the availability of a subsidy — is critical 
to placing them in jobs. The fact that a substantial number of OJT parti- 
cipants were placed through their own efforts in TOPS raises two concerns. 
On the one hand, it suggests that many of the TOPS participants may have 
been employable even without the subsidy, in part because they had gone 
through the preliminary job preparation stages of the program. For such 
participants, the subsidy may simply represent an employer bonus. On the 
other hand, it is possible that more intensive staff efforts would have 
resulted in higher OJT placement rates, particularly among harder- to- employ 
participants. The use of client job search does, however, alleviate seme 
of the concerns that an OJT program may prolong the period of welfare 
receipt and delay job placement because participants are waiting for staff 
to find them a job rather than making efforts on their own behalf. 
2. Characteristics of OJT Participants 

Who, in fact, participated in TOPS OJT, through either staff 
efforts or their cwn efforts? This question is interesting given the 
typical assumption underlying OJT placement efforts that the less qualified 
individuals are harder to place because of enployer preferences. The 



demographic characteristics cf the TOPS OJT participants shed sane light on 
the issue, although the small sample and lack of data on motivation or work 
maturity render this suggestive rather than conclusive evidence. 

What the data indicate is that despite staff concerns that not having 
a high school education would hurt participants' employment efforts, a 
substantial proportion of enrollees of all educational levels were placed 
in OJT positions. (See Table 3.5.) Further, the placement rate for the 
more educated individuals was not significantly higher than that for the 
less educated ones. 

The fact that 57 TOPS experimental s never participated in OJT because 
they moved directly into unsubsidized employment within the 12-raonth 
follow-up period suggests that the group which did not get OJT may have 
included the most job-ready candidates. (See Table 3.1.) Interviews with 
WEET and JTPA staff also indicate that some TOPS participants were 
reluctant to take jobs that paid enough to move them off the welfare rolls 
but left them below the poverty level. According to staff, this concern 
featured prominently in the pre-vocational training discussions of 
appropriate job paths and household budgeting. 

3 « Characteristics of the OJT Positions 

Although TOPS was targeted to private sector employment, 27 
percent of the jobs were in the public or non-profit sector. (see Table 
3.6.) This reflected both the rollover from work experience positions in 
the public or non-profit sector, and the fact that jobs in hospitals or 
government installations were appealing because they offered relatively 
higher wages and good benefits. 

The OJT positions in TOPS paid, on average, more than the minimum 
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TABLE 3,5 
MAINE 



ON-THE-JOB TRAINING PARTICIPATION RATES FOR EXPERIMENTALS, 
BY SELECTED CHARACTERISTICS 



Character 1 s tic 


Ever Perticlpeted In OJT [%] 


Total Sample 


3D. 6 


Age 

24 Years or Less 

25 to 34 Years 
35 to 44 Years 
45 to 69 Years 


31.1 
2B.9 
35.0 
33.3 


Ethnl cl ty 

Whi te, Non-Hispenlc 
Black, Non-Hi speni c 
Hispanic 
Other 


31.5 
20.0 
D.D 
D.D 


Degree Received 
None 

Generel Equivalency Diploma 
High School Diploma 


26.7 
3b. 3 
31.0 


Marital Status 
Never Merrled 

Married, Living With Spouse 
Merried, Not Living With Spouse 
Divorced or Widowed 


36.4 
D.D 
26.2 
29.9 


Any Children 

Less Then 6 Years 
Between 6 end 18 Years 


27.6 
31.2 


Prior AFDC Dependency 
Two Years or Less 
Mere Then Two Years 


27.2 
33.2 


WEET Status 
Monde t o ry 
Vo I un ta ry 


30.8 
30.7 


Employment Status During Quarter 
Prior to Rendom Assignment 8 

Empl oyed 

Not Employed 


32.6 
30.3 



[continued ] 
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TABLE 3.5 (cont< nued) 



Lha r act a r i st 1 c 


Ever Participated in OJ T (X] 


Employment Status During Four 
Quarters Prior to Random Aesigmment 8 

Em pL oy ad 

Not Employed 


26.8 
32.5 


Sample S1ze b 


297 


SOURCE; MDRC CaLcuLatlons from MORC Client Information Sheeta, WEET 
Information System date? end JTPA Information Syetem data. 



NOTES: ALL aembere of the seraple ere femele singLe h ee ds- o f-h o u sehol d . 
e 

CeLcuLated from Unemployment i.ieurence earnings records from 
the State of Mbi.ie. 

b 

For selected ch ar act e r i st 1 ce» sample sizes mey vary up to 10 
saapla pointe due to miesing deta. 
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TABLE 3.6 
MAINE 

G ELECT ED CHARACTERISTICS - TOPS ON-THE-JOB TRAINING POSITIONS 



Cha racteristic 



Sector of Position (%) 
Pri va ta 

Publlc/Non-Prof i t 

Hourly Wage Rate 1%) 
Less Than S3. 35 
$3.35 

$3.36 - $4.00 
$4.01 - $5.00 
$5.01 o? More 

b 

Average Hourly Wage (S) 

Scheduled Length of On-the-job 
Training Contract (%) 
6-10 Weeks 
11-15 Weeks 
16-20 Weeks 
21-25 Weeks 
26 Weeks 

Average Scheduled Length of On-the-Job 
Training Contract (Weeks) 

Actual Duretion of On-the-Job 

c 

Trei ni ng Po si t i on 1%) 
5 WeekB or Lees 
6-10 Weeks 
11-15 Weeks 
16-20 Weeks 
21-25 Weeks 
26 Weeks 

Average Duretion of On-the-Job 
Training Position (Weeks) 

Hours Scheduled to Work Per Week (X) 
20 or Less 
21-30 
31-39 
40 



Average Hours Worked Per Week 



73.0 
27.0 



1.1 
B.7 
57.6 
26.1 
6.5 



4.04 



12.0 
21 .7 
16.3 
13.0 
37.0 



19.2 



B.7 
14.1 
26.1 
13.0 

9.B 
26. 3 



16.5 



5.4 
15.2 
14.1 
65.2 

36.4 



[conti nued ) 
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TABLE 3 ,6 (continued ] 



P h Q r* a f* ^ a n ^ p V ^ #» 
Olid rod la r l b U (< 




Inh PI flccif { r>of{ nn f 1 
JDD wlabol I 1 Co tl Oil I * J 




Clerical Occupations 


46.7 


oecra ca ry p Aami n i strati ve Clerk 


16.3 


Receptionist, Telephone Operator 


6.5 


Billing, Stock, or Mail Clark 


8.7 


Clerk Typist, Keypunch Operator 


5.4 


Bank Tallar 


3.3 


C o m n ii t". ft r flnppnfnp 


2.2 


D n n L L' a o ft a i* 

oooKKaepar 


2.2 


Clipping Service Reader 


2,2 


Service Occupations 


25.0 


Food Service 


13.0 


Salesperson, Cashier 


6.5 


Nurses Aide, Haelth Care Aide 


3.3 


Child Care Worker 


2.2 


Manual Skill Occupations 


10.9 


Mach i ne Tredes 


6.5 


Seamstress, Tailor 


2.2 


Electronics Assembler 


2.2 


Professional and Managerial Occupations 


8.7 


e 

Other Occupations 


8.7 


Total Number of 

On-the-Job Training Positions 


92 



SOURCE: MORC calculations from TOPS On-the-Job Training contracts, 

NOTES: Distributions may not add to 100.0 percent due to rounding. 

Job classifi cations ara based on the U.S. Oepartment of 
Leber's occ ional titles from the jHcjtlqnary of 0c c u pa t i on a j, Tj ties, 
fourth edi tiun, 1 977. 

(conti nued ] 
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TABLE 3,6 (continued] 



a 



Information on Sector of Position wds unavailable for three 
On-the-job Training contracts, 

b 

Hourly wage rates are the "starting wage" for each position. 
Some On-the-Job Training contracts specified increases over time, 

c 

"Actual Duration of On-the-Job Training Position" was 
reported to MDRC by WEET staff. 

d 

Includes credit counselor aide, recreational therapist, 
manager trainee, certified nurse's aide, aerobic dance instructor, customs 
house broker, counselor, and exercise instructor. 

e 

Includes security gua rd , microfiche duplicator, carpenter's 
its I per, laundry operator, clock repairer, production helper, furniture 
upholsterer, and floral arranger. 



Training; two On-the-Job Training contracts were written for three of these 
participants. Two expe r i me n t a I s participated in On-the-Job Training under 
JTPA and are not included in these calculations. 



Eighty-nine ex pe r i me n ta I s perticipeted i 



n TOPS On-the-Job 
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wage. 10 The average hourly OJT starting wage was $4.04 — just above the 
TOPS target of $4.00 per hour, it is important to note that wages in Maine 
are generally low and wages in traditionally /"female* occupations ~- where 
most of the TOPS jobs were concentrated -- are particularly low. About 
half the OJT positions were in clerical areas and another quarter were in 
service occupations. Non-traditional OJT placements — amounting to only a 
small proportion of the OJT placements — included positions such as clock 
repairer, upholsterer, carpenter's helper, print shop worker, machine 
repair trainee, and machine operators of various types. 

Employment data from the State of Maine indicate that average weekly 
wages in the retail and service trades in Maine in 1984 were $144 and $182, 
respectively. 11 on the assumption of a 40-hour work week and a minimum 
wage of $3.35, a minimum wage job would pay $134 a week; a $4.00 per hour 
job would pay $160 a week. (Almost twc^thirds of the TOPS OJT positions 
required a 40-hour work week; only a small proportion were part-time jobs.) 
Thus, the starting wages of the TOPS participants in OJT seem to be in line 
with the general wage levels in these occupational fields in the state. 
They are also comparable to the average starting wage for AFDC recipients 
placed by WEET and TOPS in unsubsidized employment in 1983 and 1984. 12 

A review of the job descriptions in the OJT contracts indicates that 
the training potential of most of the positions was limited; generally, the 
positions appeared to offer the type of training associated with familiar- 
izing any new employee with the routines of a particular work setting. The 
OJT jobs also appeared to provide little opportunity for advancement. For 
example, 13 percent of the positions were for jobs as short order cooks, 
counter help, food preparers in commercial establishments (some of them 
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fast food restaurants) or institutional residences. Of the clerical 
positions, more were clerk/typist, receptionist, and similar positions than 
more demanding secretarial jobs, although one woman was training as a legal 
secretary and another was in a bookkeepers position. 

Under JTPA rules, the length of the OJT was tied to the wage level — 
the higher the wage, the longer the permissible training period, up to the 
six-month maximum. WEET practice was more flexible on training length. 
The average scheduled duration of the OJT positions was 19 weeks; 37 
percent were written for the maximum length of six months or 26 weeks. 
4 . Monitoring the OJT Positions 

Once TOPS clients were placed in an OJT position, they had less 
contact with program staff than in the earlier components, and were treated 
much like other working clients by WEET and JTPA staff. The OJT positions 
were generally monitored by the JTPA job developers. This typically 
consisted of a monthly telephone call to the employer or the participant; 
in a few locations, staff were scheduled to make one or two visits to the 
worksite during the OJT period. in general, WEET caseworkers left it to 
the OJT participants to contact them about any problems they were 
experiencing. 

Provision of WEET support services also declined early in the OJT 
phase. WEET provided child care assistance and transportation money only 
during the first month of the OJT, and a number of staff indicated they did 
not like to pro/ide the maximum during that period because they did not 
want clients to become too dependent on it. After the first month, child 
care had to be covered by the Social Services Block Grant funding available 
through Title XX of the Social Security Act or other sources; slots paid 
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through Title XX funds were not always available when clients needed them. 
Medicaid eligibility, however, continued throughout the OJT phase, even if 
earnings reduced the AFDC grant to zero. 

5. Client Performance in OJT 

Two important criteria of success in OJT are whether clients 
retain their positions for the duration of time stated in the contract, and 
whether they ;roll over," into permanent unsubsidized jobs with the same 
employer. State-reported data from Maine indicated relatively high comple- 
tion and rollover rates for the TOPS participants who were placed in OJT 
positions: 69 of the 09 individuals for whom OJT contracts were written 
completed clu> terras of the contract, and 63 remained in employment with the 
same employer after the OJT period had ended. (See Table 3.7.) Those who 
were retained generally received a wage increase. 

Corpared to interim state-reported data from grant diversion funded 
OJT programs in other states, the rollover and completion rates in the OJT 
component of TOPS appear to be relatively high. 13 These other programs 
differed from TOPS, however, in that although they placed less emphasis on 
targeting clients with previous welfare experience and little job experi- 
ence, they also did less screening for appropriateness and did not provide 
the intensive job-preparation training before OJT placement, staff in the 
other states believed that the lack of post-placement counseling and other 
support services lowered the completion and rollover rates in OJT. 

Information on the 72 TOPS OJT contracts that were not completed indi- 
cated that 13 participants voluntarily quit their jobs? five were fired? 
two moved to another area? and two were in businesses which folded or laid 
staff off. Situational problems played a large part in participants' 
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TABLE 3.7 



MAINE 



CONTRACT OUTCOMES 
FOR TOPS ON-THE-JOB TRAINING PARTICIPANTS 



Performance Indicator 


Numbe r 


Pe rce n t 


Completed en On-the-Job 

8 

Tre 1 ni ng Con tr ec t 


69 


77.5 


Continued in Employment with en 
On-the-job Treinlng Employer 


63 


70.8 


Perticipeted 1n TOPS On-the-Job 
b 

Tre 1 n1 ng 


89 



SOURCE: HDRC calculetions from WEET Information System detrs end 
reports to HDRC by KEET staff. 

e 

Three of these contrects ere for individuals who also had en 
On-the-Job Training contrect *h 1 ch they did not complete. 

b 

Eighty-nine exp e r 1 me n t e I s perticipeted in TOPS On-the-Job 
Training. Two experimented perticipeted in On-the-Job Training undar JTPA 
and ere not included in these calculetions. 
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inability to finish the training period according to schedule. The 13 
individuals who quit their jobs gave the following reasons for their 
action: six had problems with child care, transportation or health; two 
were experiencing family or emotional problems; and four were dissatisfied 
with the job. One quit because she thought she was not doing well enough, 
although the employer had no complaints. Most of the participants who did 
not finish the contract period had been working for sane weeks when the job 
ended: the average was 8.8 weeks; the range was between one and 21.5 
weeks. 14 

More about the qualifications of the TOPS participants and their per- 
formance on the job can be learned from interviews with the OJT employers. 
Interviews conducted by MDRC staff with 14 supervisors ot participants 
placed in CUT between May and November 1984 indicate that most participants 
were judged to be as productive as any new employee, and were generally 
rated as high or higher than the average new employee on attendance and 
punctuality, dress, ability to accept criticism and complete tasks as 
directed. 15 The OJT supervisors rated the TOPS employees as adequate or 
more than adequate in reading, writing and arithmetic skill, when they 
started working, but a minority of the participants were judged inadequate 
in general skills like communication, cooperation and working without 
supervision. Ten of the 14 supervisors said the work required specific 
occupational skills, and three found the TOPS employees lacking in these 
skills when they began work. Those participants judged adequate as well as 
those judged inadequate improved their communication and occupational 
skills during the training period. 
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6 . The Administration of Grant Diversion Funding 
The availability of the diverted funds from AFDC grants was an 
important source for covering the cost of the subsidized OJT wage in TOPS. 
It had been estimated — based on assumptions of participants' likely 
earnings, family size, deductible expenses and the state's standard of need 
— that the diverted funds would cover the 70 percent of the employer 
subsidy for which WEET was responsible. In actuality, grant diversion 
funded only 51 percent of the cost of the subsidized wage. 16 (See Table 
3.8.) Regular WEET funds were used to fill the gap. 

Maine staff involved in processing the grant diversion wage pool and 
reimbursing employers described the mechanics of grant diversion as a 
"bookkeeping process." no more complicated than other accourting procedures. 
There were, however, sane problems in the administration of the wage pool 
during tho first year of grant diversion. Some of these were particular to 
the procedures set up in Maine, where the AFDC benefit records and the 
grant diversion accounts were not part of the same automated system. 
Others pro/ide useful caveats for all states interested in operating grant 
diversion. 

A particularly serious problem restated from the practice of beginning 
grant diversion at the start of the OJT job. Because of retrospective 
budgeting, the AFDC grant level was not actually affected until two months 
after the new earnings began? thus, by diverting the funds in the first 
months of the OJT job, relatively little money was diverted into the wage 
pool. To correct for this, staff developed a system to divert the funds 
during the ; effective grant diversion period, R which they defined as 
beginning two calendar months after the start of the OJT job, and 
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TABLE 3.8 



MAINE 

GRANT DIVERSION CALCULATIONS 
FOR TOPS ON-THE-JOB TRAINING PARTICIPATION 



Total Calculated Wage Costs [I] 8 




223 


,135.90 


Total Calculated Employer Subsidies 


i*) b 


111 


,587.85 


Total Grant Oi verted Amount {$] 




57 


,050.00 


Contracted Grent Oiversion Coverage 
Employer Subsidies [%) 


of 




70.0 


Actuel Grant Oiversion Coverege of 
Employer Subsidies [%) 






51 .1 


c 

Sera pi e Size 


89 



SOURCE: HORC celculetions from TOPS On-the-Job Treining contracts, WEET 
grent diversion records, end TOPS opereting contrect between WEET end JTPA. 

a 

NOTES: Wage costs are celculeted es On-the-Job Training contract 
starting wage multiplied by 0 n- th e-J ob . Tr ei ni n g contrect scheduled hours per 
week, the product of which is multiplied by actual length-of-stey (in weeks] 
according to WEET progrem staff records. 

b 

Employer subsidies ere celculeted es 5056 of wege costs. 

c 

Eighty-nine e xpe ri me nte L s perticipated in TOPS On-the-Job 
Training. Two expe rimentel s perticipeted in On-the-Job Treining under JTPA end 
are not included in tr-.se celculetions. 
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continuing for two months after the OJT period ended, (These and other 
issues are discussed in detail in Appendix B.) 

***** 

The TOPS planners had anticipated that the sequence of activities 
provided in TOPS would enable the participants to obtain better-quality 
jobs — jobs paying higher than the minimum wage, in the primary labor 
market, and offering opportunities for stable employment or advancement — 
that would enable the new employees to leave the welfare rolls. It was also 
hoped that the OJT placements would include positions in non-traditional 
jobs for wonen. As discussed, sane but not all of these criteria were met 
in the OJT placements. whether the jobs were better than TOPS enrollees 
could have gotten without TOPS training can only be answered by comparing 
the employment and earnings experiences of the experimental with those of 
the controls, an issue addressed in Chapter V. Chapter IV prepares for the 
discussion of TOPS impacts by describing the experiences of the control 
group. 



CHAPTER IV 

ENVIRONMENT OF THE PET ^ RATION AND THE 
CONTROL GROUP > , j>ERIENCE 

The experiences of individuals given the opportunity to participate in 
a program like TOPS can only be appropriately evaluated in the context in 
whiwn they occurred. Two dimensions are particularly relevant. The first 
is the economic and service environment in which the demonstration took 
place, in particular, the impacts of welfare employment programs can have 
very different implications depending on the labor market conditions and 
the AFDC eligibility and benefit provisions during the period of the 
demonstration. 

The second dimension is what the experiraentals would have done in the 
absence of the program. Research confirms that there is substantial 
turnover among the AFDC population -- in and out of the labor market and on 
and off the welfare rolls — that is unrelated to a particular program 
activity. 1 There are also employment- related activities available that 
have no connection with the program being evaluated. The experience of the 
control group, as noted in Chapter n, is used as the benchmark to measure 
the effects of normal caseload turnover and the availability of other 
ewployment-related activities in the absence of the demonstration. 

This chapter begins by providing an overview of the labor market 
conditions and AFDC eligibility and benefit provisions in Maine during the 
demonstration period. It then discusses the employment-related activities 
in which controls participated, and their employment and welfare experi- 
ences. This discussion provides the context for Ihe impact results 




presented in Chapter V. 

I. The Environment of the Demonstration 

The 1980s saw changes in both the economic and welfare policy environ- 
ment in Maine. In 1980, the unemployment rate in Maine was somewhat higher 
than the national average (7.8 versus 7.1 percent), the median family 
income in Maine was lcwer than the national average ($16,167 versus 
$19,587), and more of Maine's population was living below the poverty level 
than in the nation as a whole (almost 15 versus 13 percent). 2 Between 1980 
and the start of the demonstration in 1983, the unemployment rate rose in 
Maine, but less rapidly than the national rate, and the proportion of 

persons living below the poverty level in Maine dropped despite a rising 

3 

rate nationwide. Conditions were improving in Maine relative to the 
national picture. 

A. AFDC Eligibility 

The AFDC program in Maine has been characterized historically by a 
higher-than- average standard of need, average benefit levels and a 
relatively high proportion of AFDC recipients combining welfare with work. 
In March 197 9, for example, 23.2 percent of the Maine caseload worked and 
received welfare, compared with 13 percent nationwide. 4 Passage of the 
Omnibus Budget Reconciliation Act (OBRA) of 1981 led, both in Maine and 
nationally, to a reduction in the proportion of the AFDC caseload that 
worked. The proportion of Maine welfare recipients who worked, for 
example, had dropped to 12.6 percent in January of 1983. 5 

Three OBRA provisions affected the work incentives in the AFDC 
program. One imposed a gross income test for AFDC eligibility, making 
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families with income exceeding 150 percent of a state's need standard 
ineligible for AFDC. In addition, QBRA changed the incentive to work by 
capping child-care deductions and standardizing the allowable deductions 
for work expenses at $75 for full-time workers and $38 for part-time 
workers. The latter change was beneficial to working recipients with work 
expenses below the standard, but detrimental to those with higher expenses. 
OBRA also changed the income disregard provisions for working recipients. 
Prior to OBRA, the following items were subtracted from total earnings to 
calculate countable earnings: the first $30 of earnings; one-third of the 
remaining earnings; and then deductions for child care and work expenses. 
A recipient with earned income, therefore, could keep a certain proportion 
of any earnings. OBRA eliminated the .'thirty and a third' disregard after 
four months of earnings and made the one- third disregard apply to net 
rather than gross income. These changes reduced the financial incentives 
for AFDC recipients to work. 6 

Maine responded to these changes by modifying its grant calculation 
procedures to allow recipients to keep more of their income. The maximum 
grant in Maine has traditionally been set at 72.5 percent of the AFDC need 
standard. Prior to 1983, countable income was subtracted from this maximum 
grant level and the difference was the AFDC benefit amount. 7 However, 
since 1983, instead of being subtracted from the maximum grant (72.5 per- 
cent of need), countable income is subtracted from the full need standard 
(i.e., 100 percent of need). The grant amount is either that difference or 
the maximum grant, whichever is lower. The net result of this change is to 
permit recipients to have countable income up to 27.5 percent of the 
standard of need before losing any benefits. Maine is one of only a few 
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states which calculate AFDC payments in this manner. The effect is to make 
it more likely that individuals will mix work and welfare. 

At the beginning of random assignment (October 1983), the standard of 
need for a family with one child ~ the size of the typical TOPS sample 
members family — was $349 and the maximum grant for such a family was 
$243 . 8 By January of 1987, the standard of need for a family with one 
child had increased to $416 and the maximum grant — still 72.5 percent of 
need ~ was $301. 9 Compared to other states, Maine's standard of need 
remained relatively high throughout the demonstration period. By 1986, it 
was the 14th highest in the nation. 10 

B. Labor Market Conditions 

During the demonstration, unemployment was decreasing in all areas of 
the state; the state's unemployment rate dropped from 9 percent in 1983 to 
5.3 percent in 1986. The unemployment rate for women remained just below 
the state- wide rate throughout this period. 11 Important differences in 
regional and seasonal employment persisted, however? the rate in the 
northern part of the state (Region V) ranged from 5.7 to 12.5 percent 
compared to a 2.3 to 5 # 7 percent range in the more urban southern region 
(Region I) . 

The minimum wage in Maine also rose a number of times during the 

demonstration period, as noted in Chapter III. At the beginning of randan 

assignment, the minimum wage was equal to the federal minimum of $3.35 per 

hour. On January 1, 1985 , the state minimum wage was raised to 10 cents 

above the federal level; on January 1, 1986, it was raised another 10 
12 

cents. " Since the end of the TOPS demonstration it has been raised ctill 
higher. 
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II* Employment and Training Activities of Controls 

TOPS sample members were recruited and assessed by the WEET worker who 
had caseload responsibility for them. (WEET caseloads were allocated to 
case workers either geographically or alphabetically.) WEET staff were 
permitted to tell control group members about alternative services avail- 
able through WEET and other agencies immediately after informing them of 
their research status. From this point, TOPS controls were treated like 
any other WEET clients who showed interest in participating in employment- 
related activities. Controls were eligible to receive services through 
WEET, JTPA, or other local programs, including individual TOPS-like 
activities though not the full TOPS sequence and not as part of a special 
group, staff assessed the controls' employment needi, provided counseling, 
referred them to other training or education providers, and helped to 
arrange employment or training opportunities and financial assistance or 
other support services. 

Rates of participation in employment-related activities of controls 
are compared with those of experlmentals in Table 4.1, as reflected in 
participation records from the automated tracking systems of WEET and JTPA. 
It is worth stressing again that, unlike other MDRC welfare employmert 
evaluations, the TOPS evaluation focuses on a new program within the 
state's WIN Demonstration system, not the whole system. In the TOPS 
evaluation, the control group members are the ones receiving the regular 
WIN Demonstration services. 

Just over half (52 percent) of the control group participated in some 
employment-related activity during the full follow-up period. This parti- 



-n- 1 * V 



TABLE 4.1 
MAINE 



PARTICIPATION IN WEET AND JTPA SERVICES 
OVER THE ENTIRE FOLLOW-UP PERIOD, BY RESEARCH GROUP 









Experimented 


Con t ro I e 


Pa rf o rmenco 


Ind i ca t or 


Numbe r 


Pe rce n t 


Numbe r 


Pe rce nt 


Pe r ti c i pet ed 
Activ 1 ty 


in 


Any 


276 


92.9 


76 


51.7*** 


Participated in Either 
Pre-Voca t i one l Training, 
Work Experience or 
On-tha-Job Training 


266 


89.6 


33 


22.4*** 


Perticipeted in 
Pr e-Voca ti onel 


Trai ning 


263 


88.6 


18 


12.2*** 


Pa r ti c i pe t ed 
Exper ie nee 


i n 


Work 


201 


67.7 


13 


8.8*** 


Per ti c i pa ted 
Tre i ni ng 


in 


On- the- Job 


91 


30.6 


7 


4.8*** 


Pa r t i c 1 pe t ed 
Sea rch 


in 


Group Job 


26 


8*8 


9 


6.1 


Pa r t i c i pat ed 
Job Search 


in 


Individual 


82 


31 .0 


3b 


£3.8 


Partici pa ted 


in 


Edu ca ti on 


18 


6.1 


21 


14.3*** 


Per ti c1 petod 


in 


Training 


9 


3.0 


17 


11.6*** 


Sample Size 


297 


147 



SOURCE: HORC celculetions from WEET Informetion System end JTPA Informetion 
System date. 

NOTES I Sam pie aembers heve between 11 and 26 mont,.t of participation follow-up 
dete, depending on their dete of random assignment. Thet ie, sample members randomly 
assigned early in the demonstration heve more follow-up then sample members randomly 
assigned later in the demonstration. 

Participation- is defined as ettending en ectivity for at leest one dey. 

Differences between reseerch groups ere s te t i s t i c e I ly significent using e 
chi-squere test at the following Levels: * = 10 percent? ** = 5 percent? *** = 1 
pe rcen t . 
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cipation rate is considerably higher than the participation rates of the 
control groups in other state welfare employment initiatives recently 
evaluated by MDRC — as high indeed as the participation rates of experi- 
raentals in these other evaluations - and reflects two conditions unique to 
the TOPS evaluation. First, because TOPS represented only one option among 
an array of WEET services, more alternative services were likely to be 
available to controls in Maine than in other states. Second, the recruit- 
ment and screening process made it more likely that controls would seek out 
services in Maine than elsewhere, and that WEET and JTPA staff would be 
willing to work with them. 

The TOPS control group, however, did not participate in activities to 
the same degree as experiment als, almost 93 percent of whom participated in 
at least one activity over the full follow-up period. In addition, the 
control group's activities were less intensive on average. For example, 5 
percent participated in on-the-job training, 9 percent participated in work 
experience, and 12 percent participated in pre- vocational training. These 
are considerably lower proportions than the proportions of experiment als 
participating in the various TOPS components. 

In addition, although 22 percent of controls participated in at least 
one activity similar to a TOPS component, only 3 percent participated in 
more than one, compared with 43 percent of experimentals participating in 
two, and ;>7 percent participating in all three of the TOPS components. 
(See Tables 4.1 and 4.2.) The proportion of controls who participated in 
other education and training activities was higher than among experiment- 
als; chis is not surprising given that experimentals were not supposed to 
engage in such activities except for very limited remedial education. A 
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TABLE 4.2 



MAINE 

PARTICIPATION PATTERNS IN PRE-VOCAT ION AL TRAINING, WORK EXPERIENCE, 
AND ON-THE-JOB TRAINING OVER THE ENTIRE FOLLOW-UP PERIOD, 

BY RESEARCH GROUP 





Experimentals 


Controls 


Performance Indicator 


N umbe r 


Pe r ca nt 


Numb e r 


Pe rce n t 


Participated in Pre-Voca t i o n e I Treining 
Only 


57 


19.2 


14 


9.5** 


Participated in Work Experience Only 


1 


0.3 


1 1 


7.5*** 


Pn p t i r i nn f pH in Dn- fhn- Inh Tno4 r» * n n 
ruiiiuipaiuu in u n vio juu I ro l ni ny 

Only 


0 


0.0 


3 


2.0° 


Participated in Pre-Voca ti onol Training 

cjiiu nui r ci a p c i i c 11 l> u u ii i y 


117 


39.4 


1 


0.7*** 


Participeted in Pre-Voce t i o ne I Treining 
end On-the-Job Treining Only 


8 


2.7 


3 


2.0 


Participated in Work Experience end 
On—the— Job Treining Only 


2 


0.7 


1 


0.7 8 


Perticipeted in the TOPS Sequence 
( Pre-Voce t i one I Treining, Work 
Experience, and On-the-job Training} 


81 


27.3 


0 


0.0*** 


Did Not Participate in Pr a-Voce t i o n e I 
Treining, Work Experience, or 
On-the-Job Treining 


31 


10.4 


114 


77.6*** 


Sample Size 


297 


100.0 


147 


100.0 



SOURCE: HDRC calculations from WEET Information System and JTPA Information 
Sy s tern date. 

NOTES: Sample members have between 11 end 26 months of participation 
fellow- up data, depending on their dete of rendom assignment. That is, sample 
■embers randomly assigned eerly in the demonstration heve more follow-up then 
semple members rendomly assigned later in the demonstreti on. 

Participation is defined as ettending en activity. for et least one 

dey • 

Differences between research groups are statistically significent 
using e chi-squero test et the following levels: * = 10 percent; ** = 5 percent; 
•*» = 1 perct *t. 



e 

Chi-squere test i ne p pro p r 1 e t e due to low cell frequencies. 
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sense of the relative intensity of job-related activity of the two groups 
is conveyed by comparing the per participant costs of the TOPS components 
for experimental with the per participant costs of the analogous 
activities for controls. The per participant operating cost of the work 
experience component for experimental was $390, compared with $189 for the 
work experience of controls. The per participant operating cost of the QJT 
component for experimental was $637 f compared with $301 for the OJT 
experience of controls. 13 

The trends in cumulative participation rates for controls are shown in 
Figure 4.1. Most of the control activity hz.d started by the fifth month 
after random assignment. Job search is the only activity that continued to 
grew, if slowly, throughout the observation period. 

That such a substantial proportion of the control sample participated 
in employment- oriented activities has important implications for interpret- 
ing the Impact results presented in Chapter V. The experimental-control 
differences presented in that chapter are measuring the effectiveness of 
TOPS compared to a WEET program which offered a broader array of services, 
but services that were typically less intensive than TOPS provided. 

C. Earnings and APDC Receipt Among Controls 

Figure 4 .2 shows patterns of employment and AFDC receipt -for the 
control group over the follow-up period. As can be seen, during the 
quarter of random assignment the majority of controls had no earnings and 
received sane AFDC. The next largest group had sane earnings and received 
some AFDC. That virtually all controls received some AFDC at random assign- 
ment is a reflection of the eligibility requirement that women had to have 
been on AFDC for the past six months. As the demonstration progressed, the 
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FIGURE 4.1 



MAINE 

TRENDS IN CUMULATIVE PARTICIPATION RATES FOR 
CONTROLS, BY TYPE OF ACTIVITY 
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Month of Follow-Up 

SOURCE MDRC cakiiations from WEET Information System and JTPA hfermation System data. 
NOTES: See Table 3.1. 
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FIGURE 4. 1' 
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MAINE 

TRENDS IN AFDC RECEIPT AND EARNINGS FOR CONTROLS 



Had No Earnings, 
Received Mo AFDC Payments 

Had Some Earnings. 
Received No AFDC Payments 

Had No Earnings, 
Received Some AFDC Payments 

Had Some Earnings, 
Received Some AFDC Payments 




a>* a>* ^ 



Quarter Relative to Random Assignment 



SOURCE AND NOTES: See Table C . 2 . 
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proportion of controls who had no earnings and received sane AFDC decreased 






relatively steadily. The proportion who had seme earnings and received 






some AFDC increased over the first two quarters, declined over the next 






four, then after a slight increase in quarter 7, declined through all but 






the last quarter. The proportion who had sane earnings and received no 


< 




AFDC increased relatively steadily throughout the period. The proportion 






who had no earnings and received no AF n C increased gradually over the 






demonstration, as women married, left the state or left the rolls for other 






reasons. 






These trends can be attributed to several factors. In part, they are 






due to the normal flew of welfare recipients off the rolls and into employ- 






ment. During the TOPS demonstration this flow increased with the stead'y 






improvement of the labor market. The TOPS control group (in common with 






the experimental group) was screened for motivation and potential barriers 






to participation and was, in fact, highly served. Therefore, some of the 






effect on employment and welfare receipt is almost certainly due to 






increased employ ability through activity in the other employment, education 






or training services to which control group members had access. These are 






the trends with which the outcomes for the experimentals are compared in 






Chapter V to yield the estimated net impacts of the TOPS program sequence. 
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CHAPTER V 
IMPACT OF TOPS 

The primary policy goal of the Maine TOPS program was to help AFDC 
women with histories of welfare dependency and little work experience 
obtain jobs that would pay enough to make them less dependent on welfare. 
This chapter examines the impact of TOPS on the employment and earnings of 
sample members, the extent to which they received welfare, and the amounts 
of welfare received. Chapter VI examines whether the benefits of the 
program outweighed the costs* 

The random assignment methodology of the TOPS evaluation makes it 
possible to achieve unbiased estimates of TOPS impacts by measuring 
differences in average employment and welfare outcomes between the 
experimental and control groups. 1 in this analysis, experimental- control 
differences are judged to be statistically significant if there is, at 
most, one chance in ten that the observed impact could have occurred in the 
absence of a real program effect.*" 

Before the results are presented, it is important to note several 
considerations that must be kept in mind while interpreting the results. 
First, although most TOPS sample members were on welfare and not working at 
the time of assignment, the screening process yielded a group of wcraen who 
were better educated and had fewer barriers to participation than the 
typical WIN registrant. Second, the evaluation design does not permit a 
comparison of TOPS versus no program* The control group engaged in 
substantial education- and employment- related activity. The impact 
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estimates, therefore, measure the effects of the TOPS sequence versus the 
effects of WEET and other activities engaged in by many controls. Third, 
the research sample on which the results are based includes only 444 sample 
members. Small sample sizes reduce the chances that outcome differences 
will be statistically significant. 3 

Fourth, impact estimates average net outcomes for all experimental 
including the small share of experimental who did not participate in the 
TOPS sequence. Thus, they do not measure the impacts of receiving TOPS 
services but rather of assignment to TOPS services. 4 Fifth, the impacts 
are estimated for all sample members including those with zero earnings or 
zero welfare income. 

Sixth, earnings data include OJT earnings and employment data include 
OJT employment. Seventh, as noted in Chapter II, prior earnings of 
controls were higher than those of experimental. Although the methodology 
used to estimate program impacts is designed to adjust for differences in 
characteristics at random assignment, impacts may be slightly 
underestimated. 

Finally, the earnings data were collected by calendar quarter. The 
calendar quarter during which random assignment occurred can include 
earnings before random assignment. The AFDC monthly payments data were 
aggregated into calendar quarters in order to match the earnings data. 
Therefore, the quarter of random assignment is not counted as a follow-up 
quarter for cumulative impact estimates. 

The results shew that TOPS had sustained positive impacts on earnings 
over the entire follow-up period (quarters 2 through 11). TOPS led to a 
statistically significant average earnings increase of $1,745 per experi- 



mental, or 31 percent more than average control earnings of $5,599. 

Since the earnings measure includes the earnings received by TOPS 
experimental for their OJT jobs, it is important to divide the overall 
earnings impacts into in- program and post- program impacts. During the 
period when most experiment als were in TOPS (quarters 2 through 6) there 
was no statistically significant earnings impact, although experiraentals 
did earn $555 more than controls. The earnings pattern by quarter does 
shew statistically significant earnings increases during the latter part of 
the in-prograra period, when the OJT component would have been in full 
operation. Over the period when most experimentals were no longer in the 
program (quarters 7 through 11), TOPS produced a statistically significant 
increase in average earnings per experimental of $1,190, or 36 percent more 
than average control earnings over the same period of $3,298. 

Experimentals were employed for more follow-up quarters than controls, 
4.69 quarters versus 4.21 quarters over the follow-up period as a whole. 
The difference, although reflecting an 11 percent increase for experi- 
mentals, was not statistically significant. More experiraentals than 
controls were employed in each of the follcw-up quarters except the first 
two, although most of these differences were also not statistically 
significant, perhaps because of small sample sizes. 

TOPS did not lead to reductions in welfare receipt. Indeed, more 
experimentals than controls received welfare in most quarters. These 
differences were small, however, and again not statistically significant. 

The combination of positive earnings impacts and no reductions in 
w<^fare yielded statistically significant and substantial increases in 
total measured income (earnings plus AFDC payments). Over the whole 
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follow-up period, total measured income per experimental increased by 
$1,915, or 16 percent of the control group average of $12, 198 • Over the 
ln ^^ >9raiD Peri ° d ' average mea sured income per experimental increased by 
$675, or 11 percent of the control group average? over the post-program 
period, the increase per experimental was $1,240, or 20 percent of the 
control group average. 

II* Impacts on Employment and Earnings 

This section discusses the impacts of TOPS on employment and earnings. 
Results are presented by quarter and for the full follow-up period 
(quarters 2 to 11). They are also shown for quarters 2 to 6 and quarters 7 
to 11, to separate any employment and earnings effects that include partici- 
pation in the OJT component from the employment and earnings impacts that 
occur after the program. The dividing line between the in-program and the 
post-program period is set at the end of the sixth quarter because more 
than 94 percent of the OJT contracts had ended by then and all but two of 
the remainder had ended by quarter 7. (See Table 5.1.) 

A. Employment 

Two measures of employment were used: the percentage of sample members 
who were ever employed in a quarter and during the different follow-up 
periods, and the average number of quarters sample members were employed. 
The results are shown in Table 5.2. Note that the numbers in the last 
column indicate the probability that the experimental-control difference 
could have heen observed in the absence of a real program effect. At the 
10 nercent significance level, the number in the last column is 0.100 or 
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TA8LE 5.1 



MAINE 

END OF PARTICIPATION IN ON-THE-JOB TRAINING, 
FOR ON-THE-JOB TRAINING PARTICIPANTS 



Performance Indicator 


Num be r 


Pe rce nt 


Cumu L e t i v e 
Percent 


Les t Pe r t i ci pa t ed in 








On-the-Job Training 








Quarter of Random Assignment 


0 


0.0 


0.0 


Quarter 2 


2 


2.2 


2.2 


Querter 3 


12 


21 .3 


22.6 


Que r t e r 4 


26 


29.2 


52.8 


Quarter 5 


26 


29.2 


82.0 


Querter 6 


11 


12.4 


94.4 


Querter 7 


3 


3.4 


97.8 


Querter 8 


1 


1.1 


98.9 


Querter 9 


1 


1.1 


100.0 


Que r te r 10 


0 


0.0 


100.0 


Que r te r 11 


0 


0.0 


1 00.0 


e 

Sample Size 


89 



SOURCE: HORC ceLcuLetions from WEET Informetion System dote end TOPS 
On-the-Job Treining contracts. 

NOTES: Distributions may not edd exactly to 100.0 percent because 
of rounding. 

e 

Eighty-nine of 297 e xpe r i me n t e I s participated in TOPS 
On-the-Job Treining. Two experimentel6 perticipeted in On-the-job Treining 
under JTPA end ere not included in these celculetions. 
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TA8LE 5.2 
MAINE 

IMPACTS OF THE TOPS PROGRAM ON EMPLOYMENT 



Outcome and FoLLow-Up Period 



ExperiraentaLs Controls Oifference 



81 .8 


80.2 


1.6 


0.680 


71.2 


65.6 


5.6 


0.237 


68.7 


65.3 


3.4 


0.472 


4.69 


4.21 


0.48 


0.175 


2.16 


2.00 


0.16 


0.375 


2.52 


2.21 


0.32 


0.141 


16.4 


24.7 


»8.2*» 


0.033 


23.5 


34.8 


-11.3** 


0.012 


42.8 


39.3 


3.5 


0.484 


52.1 


43.8 


8.3 


0.1 09 


49.9 


41 .3 


8.6* 


0.090 


47.9 


40,7 


7.2 


0.161 


50.4 


45.0 


5.4 


0.287 


50.2 


43.4 


6.9 


0.179 


50.6 


43 e 4 


7.1 


0.161 


50.7 


39.7 


11.1** 


0.031 


50.4 


49.3 


1.1 


0.833 



Ever Employed [%] 

Qua r te rs 2-1 1 8 
a 

Qua r te rs 2-6 
Qua r te rs 7-1 1 



Average Number of Quarters with 
Em p loy men t 

a 

Qua r te rs 2-1 1 
a 

Quarters 2-6 
Quarters 7-11 

Ever Employed [ % ] 

Quarter of Random Assignment 
Qua r te r 2 
3 
A 
5 
6 
7 



Qua r te r 
Qua r te r 
Qua r te r 
Qua r te r 
Qua r te r 
Quarter 8 
Qua r te r 9 
Qua r te r 10 
Qua r te r 11 



Sample Size 



297 



147 



SOURCE: MORC calculations from State of Maine AFOC and Unemployment Insurance 
earnings records. 

NOTES: These calculations include bC^iple members not employed and sample 
members not receiving AFOC, Experimental and control averages are regression- 
adjusted using ordinary least squares, controlling for pre-random assignment 
characteristics of sample members (sea Appendix Table C.1 and Appendix 0). There me\ 
be some di s cr op enc i e s in sums and differences due to rounding. 

A two tailed t-test was applied to aach difference between 
experimental and control groups. Statistical significance levels are provided in the 
column labeled M p" and are emphasized as follows: * = 10 percent; ** = 5 percent; 
= 1 percent. 

a 

Quarter 1 » the quarter of random assignment may contain some income 
from the period prior to random assignment and is therefore excluded from cumulative 
outcomes. 
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less. By this standard, for example, the difference in employment rates in 
the fourth quarter is near to statistical significance, 

TOPS had no statistically significant impacts either on the percentage 
of experimentals ever employed or on the average number of quarters experi- 
mental were employed over the 11-quarter follow-up period, the in-program 
period, or the post-program period. For the follow-up period as a whole, 
81.8 percent of experimentals were employed at seme point versus 80.2 
percent o£ controls, for a not statistically significant impact of 1.6 
percentage points. For both the in-program and post-program periods, the 
cumulative impacts were larger but still not statistically significant: 5.6 
percentage points on a control average of 65.6 percent for the in-program 
period and 3.4 percentage points on a control average of 65.3 percent for 
the post-program period. 

Experimentals were more likely to be employed than were controls in 
every quarter except the first two (when experimentals were likely to be in 
the pre-employment components of the TOPS sequence). In quarters 5 and 10, 
the employment impacts were statistically significant and in several of the 
other quarters the increases were not far from being significant (as 
indicated by the relatively low probabilities in the last column of Table 
5.2). In three of five post-program quarters, there employment rate 
impacts were at least 6.9 percentage points. 

As is to be expected frcm the quarterly employment- rate impacts, the 
experimentals were employed in more quarters than were controls. For the 
follcw-up period as a whole, the experimentals were employed on average for 
4.69 quarters versus 4.21 quarters for controls, yielding a not statistical- 
ly significant difference of almost half of a quarter or 11 percent. 
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During the in-program period, experimentals had 0.16 more quarters with 
employment than controls, for an 8 percent impact. During the post-program 
period, experimentals had 14 percent more quarters with employment than 
controls. Perhaps because of the small sample size, these impacts were not 
statistically significant. 
B. Earnings 

TOPS had positive impacts on earnings over the full follow-up period 
and over the post-program period. (See Table 5.3.) Over the full follow- 
up period, experimentals earned $7,3 44, or 31 percent more than controls. 
This is an increase of about $58 a month. Over the post-program period the 
experimentals earned $4,489, or 36 percent more than controls. This is an 
increase of about $79 a month. During the in-program period, experimentals 
earned 24 percent more than controls, an increase of about $555. This 
difference was not quite statistically significant at the 10 percent level. 

The pattern of earnings by quarter shows the expected reductions for 
experimentals relative to controls in the early quarters, as experimentals 
went through the pre-OJT part of the TOPS sequence — a relative earnings 
reduction that was statistically significant in the second quarter. By the 
fourth quarter, experimentals were earning $158 (or 30 percent) more than 
controls. The positive earnings impact increased in the fifth quarter to 
51 percent of the control group average; it dropped back in the sixth 
quarter to 3 0 percent, then fluctuated between about 29 and 43 percent 
through the rest of the follow-up period. Figure 5.1 summarizes employment 
and earnings outcomes by quarter. 

How are these employment and earnings patterns explained? In 
particular, what can be said about the role of the OJT component? Since 




TABLE 5.3 



MAINE 

IMPACTS OF THE TOPS PROGRAM ON EARNINGS 



Outcome and Follow-Up Period 


Expe r i me nt e I s 


Con t r o I s 


0 i f f e r ence 


p 


Average Total Eernings [$] 










Quarters 2-11 
e 


7344.00 


5599.17 


1744.83** 


0.027 


Quarters 2-6 


2855. 39 


2300. 89 


554.50 


0.1 04 


Quarters 7-11 


4488.61 


3298.28 


1190.33** 


0.020 


Averege Totel Eernings [$] 










Querter of Random Assignment 


70.97 


108.74 


-37.78 


0.1 54 


Querter 2 


172 e 65 


288.52 


-115.87** 


0.022 


Querter 3 


535.1 2 


43 6.18 


98.94 


0.267 


Querter 4 


686.94 


528.54 


158.40* 


0.078 


Querter 5 


715.91 


475.37 


240.54** 


0.011 


Querter 6 


744.78 


572.28 


172.50* 


0.093 


Querter 7 


839.99 


601 .23 


238.76** 


0.027 


Querter 8 


881 .97 


662.55 


219.42** 


0.048 


Querter 9 


917.52 


668.60 


248.92** 


0.032 


Que r te r 10 


915.73 


641 .63 


274.10** 


0.021 


Que r te r 11 


933.40 


724.28 


209.13* 


0.085 


Sample Size 


297 


147 







SOURCE ANO NOTES: See Teble 5.2. 




FIGURE 5.1 
MAINE 

IMPACTS OF THE TOPS PROGRAM ON EMPLOYMENT AND EARNINGS 




1000 r 




Quarter Relative to Random Assignment 



SOURCE AND NOTES: See Tables 5 . 2 and 5 . 3 . 



there was no separate random assignment to TOPS with and without the OJT 
component, it is not possible to disentangle the separate role OJT played 
in TOPS earnings impact s* However, Table 5*4 shows the pattern of 
participation in OJT in relation to the employment experience of 
expeUroentals* The proportion of experiraentals participating in OJT peaked 
in quarter 3 at 23 percent* The proportion participating in OJT was almost 
as high in quarter 4 at 20 percent* 

The CUT participation rates in Table 5*4 are consistent with ':he 
rising trends in employment and earnings among experimentals in quarters 3 
and 4* After quarter 4 OJT participation fell off rapidly but overall 
employment did not* Thus, as OJT contracts ended, OJT employment declined 
as a fraction of total employment. Chapter in documents that, in fact, 7 
out of 10 OJT placements rolled over into regular employment* The impact 
results are consistent with this finding* 

Tc hat extent did TOPS affect the timing of employment? Table 5*5 
sheds seme light on this issue by showing the quarter of first employment 
for experimentals and controls* AS can be seen, more control . than 
experimentals got their first jobs in the first two quarters* These, as 
noted, were the quarters when experimentals were most active in the pre-OJT 
TOPS components* Substantially more experimentals than controls got their 
first jobs in quarter 3, reflecting the effect of OJT placements* The 
results after quarter 4 show that more controls than experimentals got 
their first jobs in the later quarters* Since experimentals had higher 
employment rates than controls in the later quarters, this evidence 
indicates that they either kept the jobs they had obtained earlier or got 
other jobs* 
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TABLE 5.4 



MAINE 

ON-THE-JOB TRAINING PARTICIPATION 
ANO EMPLOYMENT STATUS 
FOR EXPERIMENTALS 



Follow-Up Period 



Quarters 2-11 
Quertere 2-6^ 
Quertere 7-11 



Quarter of Rendom 
Ab el gnmen t 
Querter 2 
3 
4 
5 
6 
7 



Al I Eicployed 



Num be r 



Pe rcen t 
of All 
Experimented 



243 
21 1 
203 



81 .8 
71 .0 
68.4 



Quarter 
Quer te r 
Que r te r 
Quarter 
Que r te r 
Querter 8 
Querter 9 
Quarter 10 
Quarter 11 



Sample Size 



47 
69 
127 
154 
146 
139 
148 
147 
148 
149 
148 



15.8 
23.2 
42.8 
51 .9 
49.2 
46.8 
49.8 
49.5 
49.8 
50.2 
49.8 
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On-the-job 
Treining 
Pertici pants* 



Num be r 



Pe rcen t 
of All 
Exp e r i me nt e I 6 



8C 
87 
5 



30.0 
29.3 
1.7 



4 
24 
68 
60 
39 
14 
4 
2 
1 
0 
0 



1 .3 
8.1 
22.9 
20.2 
13.1 
4.7 
1.3 
0.7 
0.3 
0.0 
0.0 



297 



On-the- Job 
Treining 
Perti ci pents 
as e Percent of 
All Employed [%] 



36. f? 
41 .2 
2.5 



8.5 
34.8 
53.5 
39.0 
26.7 
10.1 
2.7 
1.4 
0.7 
0.0 
0.0 



SOURCE: MORC celculetione from WEET Informetion Syetem dete end TOPS On-the-job 
Training contrecte. 

bv Pq7 TES T h e !! Ch !" C ° lUmn t "° • 8B ° bta1ned b 'y dividing the entry in column one 

■ Hi ll\ , un8d J UBt8d employment r8 te B in thie teble may differ elightly f r0 m the 

edjueted employment ratee in Teble 5.2. 

a 

Eighty-nine of 297 expe r i ma n t el s perticipeted in TOPS On-the-job Treinino 
Two experiment. is perticipeted in On-the-job Treining under JTPA end ere not included in 
these celculationB. 

b 

,h„ < * Querter 1, the querter of random assignment, may contain some income from 
the period prior to .random assignment and is therefore excluded from cumulative outcomes. 
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TABLE 5.5 



MA T N E 

IMPACTS OF THE TOPS PROGRAM ON DATES OF INITIAL EMPLOYMENT 



Outcome and Follow-Up Period 


Experimentals 


Con t ro I s 


Di f f a rence 


P 


First Employed During 










Quarters 1-11 [%] 


83.1 


81.1 


2.0 


0.593 


Quarters 1-6 [%] 


73.5 


68.5 


5.0 


0.276 


Quarters 7-11 [%) 


9.6 


12.6 


»3.0 


0.352 


Date First Employed [%) 










Quarter of Random Assignment 


16.4 


24.7 


»8.2** 


0. 033 


Quarter 2 


15.5 


18.3 


-2.7 


0.478 


Quarter 3 


21.2 


6.1 


15.2*** 


0.000 


Quarter 4 


12.9 


8.0 


4.9 


0.136 


Quarter 5 


4.3 


8.3 


-4,0* 


0.086 


Quarter 6 


3.2 


3.2 


0.0 


0.999 


Quarter 7 


1.6 


5.6 


-4.0** 


0.020 


Quarter 8 


2.5 


3.8 


-1.4 


0.434 


Quarter 9 


1.7 


0.0 


1.7 


0.112 


Quarter 10 


2.3 


1.5 


0.7 


0.613 


Qua r te r 11 


1.6 


1.6 


-0.1 


0.966 


Sample Size 


2 97 


147 







SOURCE AND NOTES: Sea Table 5.2. 
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As noted, the TOPS planners did not want the program to increase earn- 
ings only by increasing employment. They also wanted TOPS to enable the 
AFDC recipients in its target group to obtain better paying jobs than they 
would have without the program. How successful was TOPS in achieving this 
ooal? No data are available on the wage rates of sample members, but it is 
possible to look at whether TOPS changed the distribution of earnings among 
experimenuals relative to controls. 

TOPS set a target of $4.00 per hour for the wage rate to be achieved 
by those who went through the program. For individuals working 40 hours a 
week in all 13 weeks of a quarter, the TOPS target translates into earnings 
of $2,080 per quarter. Table 5.6 shows the extent to which this target was 
reached. Impacts on the proportion who earned at least the target amount 
were positive and statistically significant over the full follow-up period 
and over the post-program period. During both those periods, the propor- 
tion above the target increased and both the proportion with zero earnings 
and the proportion with earnings short of the target fell. The impacts on 
the proportion with zero earnings mirror the employment rate impacts in 
Table 5.2. Thus, although TOPS fcad little cumulative effect on the 
proportion ever employed (see Table 5.2 earlier in the chapter), it did 
affect the distribution of earnings. This finding suggests that TOPS 
achieved most of its overall total earnings impact by increasing the total 
earnings of those employed rather than by increasing the proportion 
employed. 

Comparing the employment and earnings outcomes of experimentals and 
controls who were employed provides further support for this explanation. 
Table 5.7 shews adjusted employment and earnings outcomes of sample members 
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TABLE 5.6 
HA NE 

IMPACTS OF THE TOPS PROGRAM ON THE DISTRIBUTION OF EARNINGS 



Outcome and FoLlow-Up Period 


Exp a r i men t a I s 


Con t r o I s 


D i f f e r ence 


P 


Distribution of Quarterly Earnings, 
Quarters 2-11 [%) 










Nona 


18.2 


19.8 


-1.6 


0.680 


$1-2079 


72.7 


78.3 


-5.6 


0.2D1 


$2080 or More 


9.1 


1.9 


7.2*** 


D.005 


Tot at 


100.0 


100.0 


0.0 




Distribution of Quarterly Earnings, 
Quarters 2-6 [%) 










No ne 


28.8 


34.4 


-5.6 


0.237 


$1-2079 


67,4 


63.9 


3.5 


0.471 


$2080 or More 


3.9 


1.7 


2.1 


0.237 


Tot el 


100.0 


100.0 


0.0 




Distribution of Querterly Earnings, 
Quarters 7-11 [%) 










None 


31.3 


54.7 


-3.4 


0.472 


$1-2079 


51 .2 


57.1 


"5.9 


0.250 


$2080 or More 


17.5 


8.2 


9 . 3 * * * 


0.008 


Totrl 


100.0 


100. D 


0.0 




Sample Size 


297 


147 







SOURCE AND NOTES: See Table 5.2. 
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TABLE 5.7 
MAINE 

EMPLOYMENT ANO EARNINGS OUTCOMES AMONG EMPLOYEO SAMPLE MEMBERS 



Outcome end Follow-Up Period 



Averege Number of Querters With 
Employment, If Ever Employed 

Quarters 2-11° 

Quarters 2-6 9 

Querters 7-11 

Averege Totel Earnings [S] r 

If Ever Employed 

Querters 2-11 8 
e 

Querters 2-6 
Querters 7-11 

Averege Earnings [$] , 
If ever Employed 

Quarter of Random Assignment 
Quarter 2 
3 
4 
5 
G 
7 



Quer ter 
Que r te r 
Quer ter 
Quar te r 
Quarter 
Quer ter 8 
Quarter 9 
C t;a r t e r 10 
Quarter 11 



-11 

2-6* 



a 



Sample Size 
Querterb ?, 
Quer te re 
Quarters 7-11 

Quarter of Random Assignment 
Querter 2 
Quarter 3 
Quarter 4 
x^J Querter 5 
Quartar 6 
Q <ertar 7 
Quarter 8 
Quarter 9 
Quarter 1 0 
Querter 11 



Experimental s Controls Di fference 



5.72 
3.03 
3.66 



8919. B2 
3955.70 
6463,14 



456.72 
732.70 
1235.48 
1299.94 
13.98.48 
1529.37 
1643.79 
1749.96 
1784.41 
1790.92 
1 833.33 



243 
211 
203 
47 
69 
127 
154 
146 
139 
148 
147 
148 
149 
148 



5.26 
3.06 
3.41 



7090.95 
3625.03 
521 5.96 



410.46 
829.45 
1140.42 
1254.25 
1243.28 
1472.08 
1390.82 
1546.90 
1616.60 
1657.39 
151 8.33 



118 
97 
97 
38 
52 
58 
65 
63 
63 
68 
66 
66 
60 
74 



0.46 
-0.03 
0.25 



1828.87** 

330.67 
1247.1 8* 



46.26 
-96.75 

95. 06 

45.69 
155.20 

57.29 
252.98* 
203. 06 
167.81 
133.53 
315.00* 



0.180 
0.859 
0.194 



0.041 
0 r 438 
0.051 



0.697 
0.485 
0.568 
0.724 
0.260 
0.707 
0.098 
0.193 
0.317 
0.449 
0.060 



ERLC 



SOURCE: MDRC celculetions from Stete of Heine AFOC end Unemployment 
Insurence earnings records. 

NOTES: Semple members were excluded from celculetions for periods in 
which they hed no earnings. 



See el so Teble 5.2. 
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who were employed. The experimental-control differences in this table 
cannot be interpreted as earnings impact estimates, because employed 
experimentals had different characteristics from employed controls. 
However, the pattern of earnings differences between the two employed 
groups does shew that earnings for employed experimentals were higher than 
for employed controls. The net earnings impact reached one peak in quarter 
5 and another in quarter 1C , (See Table 5.3.) Among the employed group 
the differences in the earnings of employed experimentals and employed 
controls followed a similar pattern. This does not necessarily imply 
higher wages, of course. The experimentals may have worked more hours per 
week or more weeks per quarter than controls. 

The discussion so far indicates that the impact of TOPS on earnings 
comes in part from an increase in the proportion ever employed and in part 
frcra an increase in the number of quarters with employment for those who 
were ever employed. Ideally, it would be important to know whether TOPS 
also increased the duration of employment for experimentals in the months 
they worked, whether TOPS increased hours per week, and whether TOPS 
increased hourly wage rates. Unfortunately, this is not possible because 
no information is available on hours per week, weeks of work or wage rates. 
It is possible, however, to estimate how much of the total impact on 
earnings is cue to (1) an increase in the proportion ever employed during 
the follow-up period, (2) an increase in the average number of quarters 
with employment for those ever employed during the follow-up period, and 
(3) ,ner average earnings per quarter with employment. 

Table 5.8 shews the proportions due to each of the three possible 
causes. Note that the third source of earnings impacts is some combina- 
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TABLE 5.8 



HAINE 



DISAGGREGATION OF CUMULATIVE 
EARNINGS IMPACTS 



FoL Low-Up Perl od 


Proportion of Impact 
Due to Difference 
1 n CumuLa t1 va 
Employment Retee [%] 


Proportion of Impect 
Due to D1 f f erence 
I 1n Number of Quarters 
With Employment [%] 


Proportion of Impect 
Due to Dlffert.tce In 
Averege Earnlnge Per 
Employed Quarter [%] 


Quarters 2-11 8 
Que?tere 2-6 8 
Quarters 7-11 


7.1 
34.2 
11 .5 


18.6 
-14,2 
18.5 


74.4 
80 .0 
70.0 



SOURCE: MDRC celculetlons from Stete of Maine AFDG end Unemployment Influence 
eernlnge recorde. 

NOTES: The cumuletlve eernlnge Impecte d1 ee g gre ge te d here were preeeR**d 1n Teble 5.3. 
The method for this d1 ee ggre&e t1 on 1e explelned 1n Appendix D» 

Beceuse of dete limitations* the proportion of the cumuletlvs eernlnge Impect due to 
experlmentel-control dlfferencee In edjueted eernlnge per querter with employment mey not be 
ett;r1buted entirely to dlfferencee 1n hourly wege retee. Dlfferencee 1n houre of work per week 
end differences 1n number of weeks of work psr querter with employment are additional sources 
of the difference 1n edjueted eernlnge per querter with employment. 

See el eo 1 ebl e 5 .2 • 
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tion of higher wages per hour, more hours of work per week and more weeks 
of work per quarter t it is not possible to differentiate among these given 
available data. Aside from rounding error, the proportions across each row 
of Table 5.8 add up to 100 percent, i.e., the total cumulative earnings 
impact. The second component in the second row is negative because the 
experimental-control difference in number of quarters with employment was 
negative over quarters 2 through 6. (See Table 5.7.) 

Over the full follow-up period, 7.1 percent of the earnings impact was 
due to the increase in employment rates, and 18.6 percent was due to an in- 
crease in the number of quarters with employment among those ever employed. 
Three-quarters of the earnings impact, therefore, was due to more earnings 
in the quarters employed, i.e., a combination of higher wages per hour, 
more hours per week and more weeks of work per quarter employed. Increased 
earnings per quarter with employment was the most important source of the 
earnings gain in the in-program and post-program periods as well as in the 
follow-up period as a whole. 

III. Impacts on Welfare 

The intent of Jie TOPS sequence was to reduce receipt of welfare among 
the experimental group by increasing employment and earnings. This section 
examines whether the TOPS earnings impacts did indeed reduce welfare 
dependency. It should be remembered that the estimated impacts on welfare 
payments discussed here reflect only the AFDC dollars paid directly to the 
client — not the part of the grant that was diverted to a subsidy pool for 
the OJT positions. Estimates of the effect of TOPS on overall welfare 
expenditures are addressed in Chapter VI. 
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A. AFDC Receipt 

TOPS did not reduce welfare receipt over the full follow-up period, 
the in-prograra period or the post-program period. Table 5,9 shows the 
percentages of both experiment als and controls who ever received welfare 
and the average number of months of welfare receipt. Over the follow-up 
period as a whole almost all sample members (98 percent) received AFDC 
payments at seme point. This is to be expected, given that being on AFDC 
was a requirement for eligibility. The percentages of experiment jls and 
controls who ever received AFDC during the in-program period were the same 
as over the whole follow-up period — again a reflection of the eligibility 
criteria. The percentage who ever received AFDC during the post- program 
period declined for both experimentals and controls. The percentage on 
welfare declined more for controls than for experimentals; however, this 
difference is not statistically significant. The quarter-by^-quarter 
experimental-control differences tell essentially the same story- There 
was no statistically significant TOPS impact on welfare receipt in any 
quarter, and for the post-program quarters more experimentals than controls 
received welfare. 

TOPS did not reduce AFDC payment amounts either. (See Table 5.10,) 
For the follow-up period as a whole experimentals received $6,768 in AFDC 
payments, on average, or 2.6 percent more than the control average of 
$6,5 99. Over the in-program period, experimentals averaged $3,871 in 
welfare, payments, or 3.2 percent more than the control average of $3,7 50. 
For the post-program period, experimentals received $2,898, or 1./ percent 
more than the control average of $2,848. in no quarter was the payment 
difference between the two groups more than $45, and in all but three 
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TABLE 5.9 
MAINE 

IMPACTS OF THE TOPS PROGRAM ON AFOC RECEIPT 



Outcome end Follow-Up Period 



Expuriraentels Controls Difference 



Ever Received AFOC [%] 
e 

Quarters 2-11 
e 

Quarters 2-6 
Quarters 7-11 



Averege Kumber of Months 

Receiving AFOC 

Quarters 2-11 
a 

Que r te r s 2**6 
Querters 7-11 

Ever Received AFOC [%] 

Quarter of Rendom Assignment 
Quarter 2 



Qua r te r 
Que r t e r 
Que r te r 
Que r te r 
Que r te r 
Que r tr- r 
Que r te r 
Quarter 10 
Que r te r 11 



9B.3 
9B.3 
73.3 



19.61 
11.39 
7,72 



98.6 
97.6 
B9.9 
B1 .6 
77.3 
73.6 
7H.0 
64.4 
62. 8 
50.9 
55.6 



96.0 
96.0 
71.6 



19.19 
11 .87 
7.32 



99.4 
96.0 
69.6 
65.1 
77.1 
71 .4 
68.7 
63.1 
60.2 
52.5 
49.6 



0.3 
0.3 
1.6 



0.42 
0.02 

0, 10 



-0.7 
-0.4 
0.0 
-3.3 
0.2 
2.4 
1.3 
1.3 
2.5 
6.4 
6.0 



0.G03 
0.603 
0.693 



0.654 
0.971 
0.469 



0.496 
0.602 
0.994 
0.390 
0.970 
0.592 
0.779 
0.769 
0.609 
0.204 
0.233 



Sample Size 



297 



147 



SOURCE AND NOTES : See Teble 5.2. 
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TABLE 5.10 
MAINE 

IMPACTS OF THE TOPS PROGRAM ON AFDC PAYMENTS 



Outcome and 


FoL Low- 


Up 


Period 


Experimented 


Controls 


Di f f e r ence 


P 


Average Totel AFDC 


Payments [ $] 










Querters 2 


-1T 






6768.45 


6598.52 


169.93 


0.660 


Quarters 2 


V 






3870 BP 


«57Cn Q e 

0 / <J U « i9 O 


*» on or 


fl A *7 c 
U « 4 / c 


Querters 7 


»11 






2897.84 


2848.16 


49.67 


0.843 


Averege AFDC 


Pa ymen t s 


[$] 










Quarter of 


Rand om 


As si gnmen t 


924.25 


941 .22 


-16.97 


C.428 


Querter 2 








918.20 


886.79 


31 .41 


0.215 


Quarter 3 








834.50 


798.73 


35.77 


0.330 


Querter 4 








741 .90 


728.16 


13.74 


0. 753 


Querter 5 








694.64 


686.92 


7.72 


0.872 


Querter 6 








681.38 


649.76 


31.62 


0.526 


Querter 7 








636.64 


636,23 


0.41 


0.994 


Querter 8 








595.98 


607.13 


-11.15 


0. 837 


Querter 9 








578.21 


582.83 


-4.62 


0.932 


Querter 10 








561 .67 


516.42 


45.25 


0.415 


Querter 11 








525.34 


505.55 


19.79 


0. 727 


Sample Size 


2 97 


147 







SOURCE AND NOTES: See Teble S.2. 
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quarters the experimentals 1 payments were slightly higher than those of 
controls, (See Figure 5.2 ♦) 

Figure 5.3 shows the trends in the proportions of experimentals in 
each of four groups over the follow-up period: chose who had no earnings 
and received no AFDC payments; those who had some earnings and received no 
AFDC payments; those who had no earnings and received sane AFDC payments; 
and those who had some earnings and received sane AFDC payments. (These 
trends are shown for controls in Chapter IV, ?igure 4.1.) The group who 
had no earnings and received some AFDC payments decreased sharply fro* 
random assignment through quarter 4, and declined slightly over the rest of 
the period. The group who combined earnings and AFDC receipt increased 
through quarter 4 and then declined for the rest of the period. The group 
who had some earnings and received no AFDC payments increased relatively 
steadily throughout the period. 

To what extent was TOPS responsible for uhese trends? Table 5.11 
shows the impacts of TOPS on the distribution of experimentals among these 
four groups in quarter 6 (the last quarter of the in-program period) and 
quarter M (the last quarter of follow-up). For both experimentals and 
controls, the proportion who did not receive welfare grew, both among those 
who did not work and among those who did. The proportion who received 
welfare shrank, both among those who did not work and among those who did. 

TOPS seems to have led to more mixing of earnings with AFDC income 
over the follow-up period as a whole. The impact on the proportion of 
people with both earnings and AFDC was 6,1 percentage points in quarter 6 
and 3.8 percentage points in quarter 11. TOPS also seems to have led to 
more reliance on earnings and less reliance on AFDC income by the end of 
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FIGURE 5.2 
MAINE 

IMPACTS OF THE TOPS PROGRAM ON AFDC RECEIPT AND AFDC INCOME 




Quarter Relative to Random Assigrment 




100 r 



FIGURE 5.3 
MAINE 

TRENDS IN EARNINGS AND AFDC RECEIPT FOR EXPERIMENT ALS 



50 



Had No Earnings, 
Received No AFDC Payments 

Had Some Earnings, 
Received No AFC ^ Payments 

Had No Earnings. 
Received Some AFDC Payments 

Had Some Earnings, 
Received Some AFDC Payments 




A** 



V 0?°' 0^ a*** Q> 
Quarter Relative to Random Assignment 



SOURCE AND NOTES: See Table C . 2 . 
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TABLE 5.11 



MAINE 

IMPACTS OF THE TOPS PROG RAM ON THE JOINT DISTRIBUTION 
OF EARNINGS ANO AFDC INCOME IN QUARTERS SIX ANO ELEVEN 



Outcone and Follow-Up Period 


Exp erf mentals 


Cont rol s 


Di fference 


P 


Employment and Welfare Status, 
Quarter 6 [%] 










Had No Earnings, Received No AFDC 
M a y me n ts 


6.3 


9.8 


-3.5 


0.200 


Had Some Eernings, Received No AFDC 
Psymen ts 


19.9 


1E.9 


1.0 


0.794 


Had No Earnings, Received Some ', ; 0C 
Paynen ts 


45.8 


49.5 


-3.7 


0.471 


Had Some Earnings, Received Some 

AFOC Pavrnonf c 


27.9 


21.8 


6.1 


0.162 


Total 


1D0.0 


100.0 


0.0 




Employment and Walferi Status, 
Quarter 11 [%) 










Had No Earnings, Received No AFDC 
Paynen t s 


15.6 


18.9 


-3.3 


0.3 93 


Had Some Earnings, Received No AFOC 
Fa/men ts 


SB. 6 


31 .3 


-2.7 


0.557 


Had No Earningsi Received Some AFOC 
Pay nan ts 


34.1 


31.9 


2.2 


0.648 


Had Some Earnings, Received Some 
AFDC Payments 


21.8 


17.9 


3.8 


0.357 


Total 


100.0 


100.0 


0.0 




Sample Size 


2 97 


1^7 







SOURCE ANO NOTuS: See Table 5.2. 
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the in-program period. The top panel of Table 5.11 shows positive impacts 
on the proportions in the two groups with earnings, and negative impacts on 
the proportions in the two groups with no earnings. This trend was less 
pronounced during the post-program period. In the bottan panel of the 
table, impacts on the proportions relying on just one source of measured 
income have changed signs; the impact on the proportion with earnings but 
no AFDC was minus 2.7 percentage points, and the impact on the proportion 
with AFDC but no earnings was plus 2.2 percentage points. 

There is no clear explanation for the lack of welfare reductions, 
given the positive impacts on earnings. Scrutiny of the AFDC data showed 
that systematic administrative errors in grant calculation were not the 
answer. Examination of a sample of case records confirmed that grant 
levels were reduced when subsidized or unsubsidized earnings existed. 

Several possible explanations seem plausible. First, the incane dis- 
regard feature of the AFDC grant calculation in Maine over the demonstra- 
tion period (discussed in Chapter IV) allowed earnings up to 27.5 percent 
of the full standard of need to be disregarded in the grant calculation. 
Any impact of TOPS on earnings in the range from 0 to 27.5 percent of need, 
therefore, would not result in grant reductions. Second, any program 
impact on earnings for persons whose earnings would have been above the 
AFDC income cutoff without the program would not have resulted in AFDC 
grant savings. Third, the pre-vocational training component of TOPS 
included discussions of how going to work would affect participants' AFDC 
grants and instruction in how participants could be their a/n advocates 
within the system. This could have led experiraentals to be more effective 
than controls in making sure that their grants were not cut, reduced or 
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eliminated inappropriately. 

Fourth, disaggregation of overall AFDC outcomes suggests that experi- 
mental* may have fallen into two groups with different patterns of effects 
on labor market and welfare outcomes. The adjusted average amount of AFDC 
income from quarters 2 through 11 was $6,768 for all experimentals. For 
the third of experimental employed in OJT slots, however, adjusted average 
AFDC income was lower than the overall average by $1,4 80. For the two- 
thirds of experimentals not placed in OJT slots, adjusted average AFDC 
income was $654 higher than the overall average. The proportion of those 
not placed in an OJT position who were employed at any time was 74 percent, 
somewhat lower than the overall cumulative employment rate of 82 percent. 
Of course, these experimental-subgroup comparisons are not impact differ- 
ences and are only suggestive that the lack of welfare savings may be 
associated with the group of experimentals who were not placed in OJT 
slots. 5 

IV. Impacts on Total Measured Income 

This section presents the overall impact of TOPS on total measured 
income, which includes AFDC payments plus earnings as reflected in 
Unemployment Insurance (UI) earnings records. As discussed in Chapter II, 
it does not include other income, such as child support payments or income 
from jobs that are not covered by UI. 

TOPS led to statistically significant increases in total measured 
income over the full follow-up period, the in-program period and the 
post-program period. (See Table 5.12.) Over the full follow-up period, 
TOPS increased measured income per experimental by $1,915, or 16 percent of 

2- 1 : 
X 0 ^ 
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TABLE 5.12 



MAINE 

IMPACTS OF THE TOPS PROGRAM ON TOTAL MEASURED INCOME 
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Qua r t e r 11 
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Sample Size 


297 


147 







SOURCE: MDRC calculations from State of Maine AFDC and Un employment 
Insurance earnings records. 

NOTES: Total measured incoma is defined as tha sum of AFDC income 
and aa rnings. 

See also Table 5.2. 
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the control group average of $12 , 198. Over the in-prograra period, the 
increase was $675, or 11 percent of the control group average of $6,051, 
Over the post-program period, the increase was $1,240, or 20 percent of the 
control group average of $6,146. The quarter-by-quarter pattern shows a 
statistically significant reduction in total measured income for 
experimental compared to controls during the first two quarters, 
reflecting greater earnings by controls as experimentals went through the 
pre-QJT components of the TOPS sequence. Total measured income was higher 
for experimentals than controls in every other quarter, and the differences 
were statistically significant. The differences grew over quarters 3 
through 5, fluctuated with no major trend over quarters 6 through 9, then 
reached a peak in quarter 10. 

For both experimentals and controls, earnings became increasingly 
important in total measured income over the experimental period. The 
trends for the two groups are shown in Table 5.13 and Figure 5.4. Table 
5.13 shows that the proportion of total measured income controls derived 
from AFDC fell steadily, from 92 percentage points in quarter 1 to 41 
percentage points in quarter 11. Except at the beginning and at the end of 
the follow-up period, this proportion was lower for experimentals. TOPS 
increased the import iice of earnings relative to AFDC income for experiment- 
als versus controls over both cumulative periods although there were no 
reductions in absolute levels of welfare payments during this period. This 
indicates there was greater mixing of earnings and welfare payments among 
experimentals. For quarters 2 through 6, the AFDC proportion of income was 
3.7 percentage points lower for experimentals; for quarters 7 through 11 it 
was 0.8 percentage points lower. TOPS also moved up the point where 
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TABLE 5.13 
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IMPACTS OF THE TOPS PROGRAM ON SOURCES OF INCOME 
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FIGURE 5.4 
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earnings overtook A?DC income as an inccme source fran between quarters 7 
and 8 for controls, to between quarters 4 and 5 for experimental s. The 
steep trend in earnings for experimental between quarters 2 and 5 shows 
1 a effect of the change fran unpaid pre- vocational training and work 
experience to paid OJT activity. 

***** 

Clearly, TOPS was successful fran the point of view of the AFDC 
recipients who were judged appropriate for the program. Whether it was a 
cost-effective program from the perspective of government budgets, or 
society as a whole, is the subject of Chapter VI. 
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CHAPTER VI 
BENEFIT- COST ANALYSIS 

This chapter presents a benefit-cost analysis of the TOPS program* 
The analysis draws on information presented in the last three chapters, as 
well as other data, in order to t\ke account of the program's varied 
benefits and costs as comprehensively as possible* In addition, these 
benefits and costs are assessed frccn several major perspectives, including 
those of taxpayers, government budgets, welfare recipients, and society as 
a whole. Thus, conclusions can be drawn about not only the overall 
efficiency of the program, but also the gains and losses to the groups it 
most directly affects. 

The analysis has many elements and, as a result, its description in 
this chapter resembles the completion of a jigsaw puzzle* Jhe first 
section establishes the puzzle's boundaries and where the puzzle's pieces 
fit within them by describing the analytical framework used* The next two 
sections examine the pieces themselves, namely the individual benefit and 
cost components included in the analysis* Assembling the puzzle is the 
task of the fourth section, which aggregates the benefits and costs *;>r 
each of the perspectives considered and then presents the overall results. 
There are unavoidably a number of missing pieces — such as benefits and 
costs of TOPS that cannot be quantified and uncertainty surrounding the 
program's long-run effects — but enough can be put together for an image 
of the program's relative effectiveness to emerge in the concluding section 
of this chapter. 
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I. Analytical Approach 

This assessment of the benefits and costs of the TOPS program uses the 
snme analytical approach followed in MDRC's previous evaluations of welfare 
employment programs, 1 although it introduces sane additional distinctions 
due to the complexity of the TOPS program. The analysis places dollar 
values on both the program's effects and its use of resources. The effects 
and uses to be considered are shown in Table 6.1, which also indicates the 
major perspectives from which they are valued and the data sources used in 
making the estimates. 

The measurable program effects include the impacts on earnings and 
welfare receipt discussed in the last chapter, as w*ll as on several other 
outcomes: fringe benefits from regular und OJT employment, tax payments, 
AFDC grant diversion amounts, Food Stamps, Medicaid, transfer program 
administrative costs, and the value of labor services provided in TOPS work 
experience and OJT assignments. In addition, the earnings impact estimates 
are disaggregated by recnilar versus OJT jobs. These additional estimates 
(of effects other than the basic earnings and welfare impacts) will be made 
using procedures discussed below. As with the welfare and earnings impact 
estimates, they represent our best estimates c the net effects of the TOPS 
proi» ram — that is, of the experimental-control differences in these 
outcomes. 

The program resource use that is valued in the analysis includes all 
expenditures, regardless of funding source, on the operation and management 
of TOPS, support services and payments to program participants, and related 
program functions such as OJT grant diversion. The primary program funding 
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TABLE 6.1 



KAINE 



EXPECTED EFFECTS TOR COMPONENTS OF THE 8B<EFIT-C0ST ANALYSIS 
BY ACCOUNTING PERSPECTIVE, WITH DATA SOURCES 
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source was WEET, which used both WIN and special demonstration program 
funds, but JTPA and other sources were also used. In addition, the use of 
non-TOPS education and training services by both experimentals and controls 
which, as indicated in Chapter IV, was substantial — will be taken 
into account. Thus, the cost estimates ultimately produced by this 
analysis reflect the net use of WEET, JTPA, and other resources by members 
of the experimental group (that is, resource use by experimentals beyond 
what was used by the control group) in the same way that the benefit 
estimates reflect net program impacts. 

Whether a given program effect or use of resources is a benefit or a 
cost depends on what is actually measure 3 and the analytical perspective 
that is taken. The plu es and minuses shewn in Table 6.1 each reflect the 
expected benefit or cost status of an item, but the actual results may be 
(and, as reported below, sometimes are) different from these expectations. 
Once measured, particular effects or expenditures will constitute benefits 
or costs, or not be relevant, depending on which of the analytical 
perspectives — welfare recipients, taxpayers, buOget and society — is 
considered. The perspective of welfare recipients iCentifiss benefits and 
costs for members of the experimental group, indicting .iow these 
individuals fared as a result of the program. The taxpayer perspective, on 
the orher hand, identifies benefits and costs from the standpoint of 
everyone in society other than these welfare recipients. For example, if a 
reduction in AFDC use were found, that effect would translate into a loss 
for welfare recipients and a corresponding benefit for taxr^yers. 

The taxpayer and welfare recipient perspectives in Table 6.1 together 
constitute the social perspective* Thus, benefit and cost result* for the 
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two groups are added together to obtain results for society* In the 
example of the AFDC reduction, the loss for one group is offset by a gain 
for the other so this transfer of money results in no net social gain or 
loss. it may also be noted that other points of view within the taxpayer 
group can be considered. One of these ~ that cf government budgets ~ is 
shewn in the table and is examined throughout the analysis. Another of 
these points of view is employers (not shown in the table, but considered 
at appropriate points in the analysis), who are obviously affected by 
programs like TOPS and hence have a stake in their performance. 

The final benefit-cost estimates for TOPS will cover a five-year time 
horizon from the point individuals entered the research sample in 1983 or 
1984. Data are available for only part of this time span — the 
observation period which generally lasted about three to four years, but 
varied in length according to the type of data and the time someone entered 
the sample. (See Table 2.4 in Chapter II.) As a result, program effects 
beyond this time will be projected over the remainder of the five-year 
period using several assumptions. Program resource use generally occurred 
during the observation period, so cost projections are not necessary. All 
final estimates are discounted — for both inflation and foregone 
investment — to reflect 1985 dollars. 



II. Program Effects 

A. Earnings and Output 

The TOPS program led to inc eased work by welfare recipients — in 
regular employment as well as in the program's work experiencr and OJT 
placements — and thus to increased output in the Maine economy. Both the 



OJT and the regular jobs provided earnings and fringe benefits to welfare 
recipients which, as expected, proved to be their principal benefit frcra 
TOPS. For the regular employment, both the value of the additional output 
and the cost of the compensation accrued to employers, so they came out 
about even. For on-the-job training, employers could do better, because 
they paid only part of the OJT wages (the remainder was picked up by WEET 
and JTPA). Finally, because no wages or fringe benefits were provided in 
the work experience positions, the full value of that output went to the 
puhlic and non-profit agencies that employed TOPS participants, and thus to 
taxpayers as well. 

As discussed in the last chapter, TOPS experimental s showed a 
sustained increase in earnings over the level received by the control 
group. The value of the increase through the entire observation period was 
$1,839 per experimental. As shown in Table 6.2, an estimated $577 of this 

difference is attributable to OJT assignments (based on OJT enrollment and 
2 

wage data ); the remaining $1,262 reflects regular employment. 

Experimentals also enjoyed an increase in fringe benefits. For 
regular employment, the value of these benefits (excluding payroll taxes, 
which are treated separately below) was esti^ted at 12 percent of 
earnings, based on national employment compensation data. 3 For OJT jobs, 
estimates were obtained frcw the worksite survey of supervisors in the 
firms where OJT participants were assigned. On average, OJT fringe 
benefits amounted to 11 percent of earnings. 

Under standard economic assumptions, the compensation paid in regular 
employment reflects the value of employees' output to employers and hence, 
barring external effects on others, to society in general. However, this 
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HAINE 



ESTIMATED EXPERIMENTAL-CONTROL DIFFERENCES IN EARNINGS, 
FRINGE BENEFITS, AND PERSONAL TAXES PER EXPERIMENTAL 
FOR THE OBSERVATION PERIOD 8 



Component of An s ly si s 


Estimate 


Earnings 




Dn^thp-.l.ih T n a 4 n 4 n n 


5 O / / 


Unsubsidized Employment 


1262 


Total 


1839 


Fringe Benefits 




On-the-Job Training 


63 


Unsubsidized Employment 


153 


Total 


216 


Ta xes 




Social Security Payroll Tax 


130 


Federal Income Tax 


194 


State Income Tax 


67 


State Sales and Excise Tax 


35 


Total 


426 


Sample Size 


444 



SOURCE: MDRC calculations from Unemployment Insurance earnings 
records and from published data on tax rates and employee benefits. 

NOTES: Differences are regress ion- adjusted using ordinary least 
squares, controlling for pre- random assignment character! sties of sample 
members. Because of rounding, detail may not sum to totals. 

a 

Th u end of the observation period was June 19 87 for 
Unemployment Insurance earnings records. 
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does not apply to TOPS work experience and OJT assignments, because employ- 
ers paid either no compensation or only partial compensation in these 
instances. The value of the output produced in these assignments is 
treated here, given assumptions that generally are more appropriate in such 
circumstances, as the full compensation employers wculd normally have to 
pay labor to provide the same output, 4 

An estime te of this cost was made using information collected in the 
worksite survey. The relative productivity of TOPS participants compared 
to employers' regular employees — estimated by their job supervisors to be 
about as productive as the regular workers on average in both the work 
experience and OJT assignments — was multiplied by the number of hours 
they worked to determine the time it would take regular workers to produce 
the same output. This was then multiplied by the appropriate hourly wage 
and fringe benefit rates for these jobs to arrU - at the full-compensation 
Intimates. 5 * 

The results indicate that the work experience positions provided not 
only job experience to TOPS participants, but also almost $700 worth of 
services per experimental to community agencies. The estimated value 
amounts to $1,314 per person who actually participated in one 9f these 
positions. in addition, the value of output produced in the OJT assign- 
ments was estimated as $652 per experimental (or $2,661 per OJT partici- 
pant), which is about the same as the OJT wages and fringe benefits paid by 
employers and the program to participants. 
B. Tax Payments 

Since TOPS produced an increase in earnings, there is a corresponding 
increase in federal and state income taxes, payroll taxes, and state sales 
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and excise taxes. The appropriate tax rates and rules were applied to 
impute taxes fran earning and other income (using experimental-control 
differences in total earnings for canputir.g payroll taxes, earnings over a 
base amount for calculating income taxes, and the corabired income from 
earnings, AFDC payments, and unemployment compensation for determining 
sales and excise taxes), 6 The estimated increases in taxes paid by sample 
members during the observation period are shown in Table 6.2. Total taxes 
increased by $426 per experimental, with federal income taxes and social 
security payroll taxes accounting for most of this amount. 

In addition, employers paid both sccial security and unemployment 
compensation payroll taxes on the increased earnings of the experimental 
group. The respective increases in these taxes were estimated as $130 and 
$26 per experimental. 

C Transfer Payments 

As described in Chapter V, experimental group members on average 
received slightly increased AFDC payments over the course of the follow-up 
period. Receipt of Me -caid, Food Stamps and unemployment compensation 
were also affected by the TOPS program, so the effects on these transfer 
programs are included in the benefit-cost analysis as well. 

For the purposes of this analysis, an experimental-control difference 
in regular AFDC payment receipt was calculated from AFDC records, as in the 
last chapter. However, the estimate of this difference shewn in Table 6.3 
cavers the entire observation period, not just the eleven-quarter follow-up 
period coumon to all sample members that was used in the impact analysis 
(as much as \7 quarters of data were available on sane sample members). 

In addition, TOPS diverted some AFDC payments to finance part of 
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TABLE 6.3 



H TNE 

ESTIMATED EXPERIMENTAL-CONTROL DIFFERENCES IN TRANSFER PAYMENTS 
ANO ADMINISTRATIVE COSTS PER EXPERIMENTAL 
FOR THE OBSERVATION PERIOD 6 



Type of Payment or Cost 


Es t i me t e 


Transfer Payments 




AFDC 




Ragula r Payments 


* 386 


Diverted Peyments 


1 92 


Unemployment Compensation 


-134 


Medi cei d 


150 


Food Stemps 


-238 


Total Transfer Peyments 


356 


Administrative Costs 




AFDC 


81 


Unemployment Compensetion 


-18 


Med i cei d 


S 


Food Stemps 


-33 


Totel Administrative Costs 


-14 


Sample Size 


444 



SOURCE: MDRC celculetions from AFDC peyments records, TOPS 
On-the-job Treining contrecte, rtete grant diversion records, Unemployment 
Insurence earnings and payments records, publish3d dete on Mediceid costs 
end welfere administrative costs, and WEET Information System deta. 

NOTES: Differences are regression-adjusted using ordinery leest 
squares, controlling for pre-random essignment chorecter i sti cs of semple 
members. Beceuce of rounding, detail mey not sum to totels. 

8 

The and of the observetion period was November 1987 for AFDC 
records, December 1986 for Unemployment Insurence benefits records, and 
Juna 1987 for Unemployment Insurence earnings records. 
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the cost of the OJT wage subsidies. Because this gran*- iiversion is an 
AFDC-funded transfer payment, an estimate of its value i_ presented in 
Table 6.3 (although grant diversion is treated as a program cost a the 
analysis, as discussed below). The average amount of AFDC diverted for OJT 
was estimated using wage information fran the OJT contracts. The average 
amount of OJT earnings pes experimental was multiplied by 26 percent which, 
according to MDRC calculations frcm program records, was the amount of the 
wage that grant diversion actually covered (i.e., 51 percent of the subsidy 
cost). As shown in the table, the restating estimate of AFDC gr^.nt 
diversion per experimental is $192 (or $639 per person who actually 
participated in an OJT position) . 

Differences in unemployment compensation were measured using UI 
records data. The overall experimental- control difference reported in 
Table 6.3 indicates a modest reduction in payments to TOPS registrants 
during the observation period. 

Program effects on the use of Food Stamps and Medicaid were estimated 
using several sources of infonr tion. Differences in Food Stamps have been 
imputed on the basis; of total measured household income — including 
earnings, AFDC, and unemployment compensation — and the earnings disregard 
and child care and medical deductions used in determining program 
eligibility ,;md benefits. 7 Differences in Medicaid were imputed on the 
basis of observed differences in AFDC receipt, rides governing Medicaid 
eligibility, and average Medicaid payments made to eligible individuals. 
An individual on AFDC, or participating in an OJT position funded oy AFDC 
grant diversion, is automatically entitled to receive Medicaid, and renains 
eligible for a specified period of time after leaving the rolls. 8 Given 



-128 



this, an experimental-control difference the number of months of 

Medicaid eligibility was estimated and then multiplied by the average 
Medicaid monthly payment ($170 per AFDC case in Maine during 1985 9 ) to 
determine the average TOPS effect on Medicaid payments. 

As indicated in Table 6,3, the program generated an overall increase 
of $165 in the value of regular transfer payments received by the experi- 
mental group during the observation period* This net increase becomes $356 
when the OJT wage subsidy funded by AFDC grant diversion is included. 
These results reflect increases in AFDC payments, grant diversions, and 
Medicaid payments. There were net reductions in unemployment compensation 
and Food Stamps, however, which partly offset the AFDC and Medicaid 
increases. 

There was a slight decrease in estimated administrative costs for 
transfer programs during the observation period due to reduced use of 
unemployment compensation and Food Stamps. The TOPS effects on transfer 
program administrative expenditures were estimated based on differences in 
usr of the transfers and state and federal program cost information. 10 

D. Future Effects 

Thus far only program effects during tiie observation period — which, 
as noted, lasted abou'- throe to four years — have been considered. 
However, these effects almost certainly vill last beyond this period, an 
expectation that should be taken into account in the analysis. Effects are 
consequently projected for each sample menber beyond what was actually 
observed, so that the measured and projected effects together cover five 
years free the time an individual entered the sample. The average length 
of the projection period is generally one to two years, but it varies by 
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sample member. For example, if the observation period for an individual's 
earnings is as little as three years, the projection period is two years. 

Projecting program effects entails selecting a base period estimate of 
experimental-control differences in the outcomes included in this analysis, 
and th^n making an assumption about future changes in that estimate. The 
base period used in this and oti.^r MDRC eva 7 uations of welfare employment 
programs is the last two quarters of the observation period. It is assumed 
that differences observed during these two quarters do not decay over the 
course of the projection period, an assumption that is consistent with the 
patterns of program impacts measured for TOPS during the observation 
period. However, this assumption differs from most other MDRC evaluations 
where, because less follow-up information wc.s available, a decay assumption 
based on other research has been used. Consequently, a sensitivity test in 
which the TOPS results are estimated using the same decay rate assumption 
used in most of the other MDRC studies of welfare employment programs (that 
is, an annual decay ra* of 22 percent) is presented later in this chapter. 

The resulting estimate., are presented in Table 6.4. The values of all 
program effects both observed and projected -- have been adjusted for 
inflation and discounted at a 5 percent real annual rate to reflect 1985 
dollars. As can be seen in the table, the projected portion of all the 
effects is smaller than the part that was actually observed. The 
sensitivity of these estimates to using alternative base, decay and 
discount rate assumptions will be tested later in this chapter. 
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ESTIMATE BENEFITS DURING THE OBSERVATION PERaOD, 
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Benef 1 1 6 

On-the-Job Training 

Uns ub s1 d 1 zed 
Emp 1 oyme n t 


652 
1218 


0 

S3 


0 

206 


0 

1531 


652 
2842 


Pe y roi I Ta x as 

Employee Portion 
Employer Portion 


118 
141 


6 
7 


13 
16 


36 
115 


220 
264 


Other Texes 


215 


60 


40 


257 


533 


AFDC Payments 
Reguler 
Di verted 


218 
195 


134 
0 


29 
0 


120 
0 


472 
195 


Other Trensfer 
Paymen 1 6 


-165 


-44 


-44 


-353 


-562 


Transfer Program 
Ado 1 n1 a tra 1 1 on 


-12 


-2 


-4 


-3 7 


-50 



SOURCE: MDRC calculations from worksite eurveyj Unemployment Insurance system 

earnings end payment records; AFDC peyments records; published dete on Mediceid 

costs, welfare edral nl stretl ve costs, tex rates, end employee fringe benef.its; and the 
WEET Information system dete. 

NOTES: Results are expressed 1n 1S85 dollars end therefore the observetion 
period estimates differ slightly from those presented in Tables 6.2 end 6.3. Beceuse 
of founding, datell mey not sum to totels. 

Besed on aveileble follow-up date. 

b 

The projection bese period 1s a quarterly overage „T the last two 
querters of aveileble follow-up for en 1nd1v1duel. Progrem effects observed during 
this baee period ace multiplied by a projection fector to estircete benefits from the 
end of the observation period to five years from the point of rendom essignment. For 
the projection estimetes 1t is assumed thet impects do not decey efter the 
observation period. 
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Ill* Program Resource Use 

A. TOPS Program Operations 

TOPS was operated irregularly ~ as a series of program cycles started 
frcni time to time in one of WEET's offices around the state — rather than 
as an ongoing program. Each cycle entailed recruiting welfare recipients 
for the program from the area served by an office and then enrolling 
appropriate individuals in the sequence of TOPS activities described in 
earlier chapters. Thus, estimating the overall operating costs of the TOPS 
program requires estimating the costs of each cycle and then aggregating 
these estimates. 

The operating cost uf each program cycle #as estimated in several 
steps. First, data were gathered from WEET and JTPA staff 01 the hours 
they spent on the cycles. JTPA staff worked on TOPS under financial and 
nonf inancial agreements with WEET. All staff time devoted to recruitment, 
assessment, pre-vocational training, work experience, OJT, and other 
program functions for the TOPS enrollees in these cycles was identified. 
Time spent on other WEET and JTPA activities or in working with other WEET 
or JTPA clients was excluded. The resulting estimates of staff hours were 
adjusted downward to eliminate activities associated with MDRC's evaluation 
research rather than program operations themselves, that is, one-third of 
recruitment and assessment time, since this served to create the control 
group, plus time spent on random assignment and cooperating with other 
evaluation needs. 

Second, these staff hours we-e multiplied by per- hour salary rates, 
narked up for fringe benefits, nonpersonnel costs, and administrative 
overhead. The rates reflect annual WEET and JTPA salaries and other costs, 



and annual staff hours data, in the appropriate offices during the time the 
cycles were operated. This yielded estimates of labor costs of operating 
each of the TOPS cycles. 

Third, the costs of WEET contracts with other agencies (most commonly 
JTPA) for services delivered during the cyclec were added, but only to the 
extent resource us* was not already captured in the labor cost estimates 
just described. For example, in sccie instances JTPA was paid by WEET for 
staff time spent on TOPS and in other cases it was not. All JTPA time is 
counted in the analysis, but it is not counted twice if WEET paid JTPA for 
the time. It should be noted, finally, that the costr* for TOPS cycles of 
allowances, support services, and OJT wage subsidies are considered 
separately below. 

The costs of the cycles were then aggregated and central administra- 
tive costs (TOPS operating costs incurred at the state level) were added. 
These latter costs were estimated using state data and interviews with 
state staff. As with the estimates for the program cycles, the research 
costs associated with central administration have been excluded. 

Table 6.5 presents the resulting estimates of the costs of operating 
TOPS in WEET's five regions in Maine. The estimates are broken down by 
region and program function. The total operating cost of the program, 
excluding research-relatec ^osts, was approximately $47 9,000. This amounts 
tc $1,613 per experimental. Alfio provided in the table are estimates of 
the operating cost for each program component per experimental and, where 
applicable, per participant in a given component. Of note are the 
relatively high costs of recruiting and assessing individuals for TOPS 
($267 per experimental) and of coordinating and managing the program 



TABLE 6.8 
MAINE 

TOTAL TOPS OPERATING COSTS, 
BY COMPONENT AND OFFICE 



WEET Regl on 


Recrul traent/ 
A o go o ora o n t 


Pr e- 

V oce ti onal 
Tr el nl ng 


Work 
Expe r 1 ence 


OJT 


Other 


To tel 


I 


$14,662 


$16,840 


$23,611 


$3,604 


$106,998 


$183,303 


II 


22,739 


6,823 


8,269 


3,396 


34,0/7 


73,296 


III 


31,441 


14,694 


13,887 


21,816 


24,1 9Q 


1 05 , 7 07 


IV 


23,372 


11,828 


14,124 


9,323 


17,940 


76,366 


V 


11,112 


26,203 


18,330 


16,819 


OR R A 0 


Jo f d U 4 


Central 


9,348 


6,892 


8,710 


4,851 


18,606 


4e^. ,o 


To to I 


112,686 


80,679 


80,802 


68,410 


226,362 


658,707 


RoaoGrch Costs 


33,143 


2,364 


2,380 


1,7 08 


40,141 


79,705 


Net Cost 


79,421 


78,226 


78,441 


68,704 


188,211 


479,003 


Cost per 
Experimental 


$287 


$283 


$284 


$191 


$827 


$1,613 


Coat por 
Per ti cl pc nt 




$297 


$390 


$637 







SOURCE: HDRC calculation from WEET and JTPA expenditure raccrda, Wormetlon gatharad In staff 
Iriervlewe, and WEET trecklna evatam data. 
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components to which these individuals were subsequently assigned (much of 
the $627 "other ■ cost). 

B * TOPS Allo wances , Support Services and Wage Subsidies 

The second category of TOPS expenditures includes allowances and 
support service payments paid with WEET funds, as ..ell as sane additional 
child care services paid for by Title XX funds. Fstimates of these 
expenditures are presented in Table 6.6. 

Child care and transportation payments were made primarily to 
participants in pre-vocational training or work experience, However, WEET 
child care was sometimes provided during the first month of an OJT 
assignment, and Title XX child care could subsequently be used. 

Other payments were made for supplies, uniforms and tools needed by 
TOPS participants for work experience positions or for regular jobs. in 
addition, during pre-vocational training and work experience, an allowance 
of §1.25 per hour was paid to cover miscellaneous personal expenses and to 
encourage continued participation. such assistance was not generally 
provided to OJT participants, because it was expected that OJT wages and 
AFDC income disregards for such work expenses would be sufficient. 
However, support was provided in sane instances. 

Da^ on WEET support service and allowance payments to all experi- 
mentals were collected by TOPS staff for the period of October 1983 through 
March 1985. Title XX expenditures during the same period were estimated 
based on information from WEET and the Maine Bureau of Social Services 
Planning and Evaluation Unit. 11 These data were used .o make the cost 
estimates provided in Table 6.6. 

The total amount spent by WEET on support services and allowances for 
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TABLE 6.6 
MAINE 



TOTAL TOPS SUPPORT SERVICE COSTS, BY COMPONENT AND OFFICE 



WEET Region 


*| " — — 


WEET 


11 ^le XX 








Allowances 


Child Cere 


Child Care 




Other 


Total 


I 




$21 ,121 


$18,381 


t^O AAA 

•13,441 


$7,961 


$5,441 


$66,345 


II 




9,129 


6,595 


3,428 




8,152 


31,213 


III 




8,554 


5»553 


c on*7 
0,oU/ 


1,786 


541 


£1,841 


IV 




10,016 


5,592 


4,925 


3,592 


1,174 


25,301 


V 




15,503 


8,014 


11*449 


3,857 


6,047 


44,869 


Total 


$64,425 


$44,136 


$38,550 


$21,105 


$21,354 


$189,570 


Average Per 
















Experimental 




$217 


C149 


$130 


571 


$72 


$638 


Average Par 
















Parson Receiving 














a Payment 




$267 


$244 


$1313 


$12VJ 


$181 




SOURCE: 


HDRC calculations 


from WEET end Title XX expenditure 


records. 






NOTES: 


The 


and of tha obser nion period 


■as March 1985 for 


support service 


coses. 
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TOPS participants was about $151,000. In addition, almost $39,000 of Title 
XX money was spent on child care for TOPS clients. On average, the total 
cost of allowances and support services per experimental was $638, which is 
considerably higher than in other welfare employment initiatives recently 
evaluated by MDRC. Child care accounted for almost half of this cost; 
allowances accounted for about 35 percent. Transportation and other pay- 
ments were a small proportion of total allowances and support service 
costs. Because payments were linked to program participation, 96 percent 
of those who participated in TOPS received at least some assistance. 

As indicated earlier, a wa<je subsidy of ^0 percent of *--he ^tal wage 
was paid to employers for on-the-job training positions. Seventy percent 
of the subsidy was paid by WEET — about three-quarters of this through 
AFDC grant diversion — and the remaining 30 percent was paid by JTPA. The 
amount of the wage subsidies was estimated from the QJT contracts, based on 
the starting wages paid, the scheduled hours per week, and the number of 
week'; worked. The estimated wag* subsidy per person in one of the 7T 
assignments was $1,254, including $639 in grant diversion. 

The total expenditures on the TOPS program — both the operating 
expenses presented in the previous section and the allowances, support 
services, and subsidies discussed in tius one — are srjmarized in Table 
6*7. The total aggregate expenditure on the program, excluding estimated 
research costs, was about $7 80 ,000 . This amounts to $2,627 per experi- 
mental, or $2,933 per person who participated in any TOPS activity. WEET 
plus grant diversion paid for approximately 80 percent of this expenditure, 
with JTPA covering 15 percent and other funding sources covering the 
remaining 5 percent. 
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TABLE 6.7 
MAINE 

TOTAL TOPS COSTS BY SOURCE OF FUNDING AND OFFICE 





WEET Costs 












fc*jf*f*^ O mm. mm* Z 

WEET Region 


Opera A ng 


Support 
Services 


JTPA 

e 

Costs 


Title XX 
Chi Id Care 


Other 
Agency Costs 


Wage 
Subsidies 


Total 


I 


$156,642 


$52,904 


$6,460 


$13,441 


0 


$31,377 


$261,025 


II 


60,374 


27,785 


12,392 


3,423 


$531 


15,044 


119,553 


III 


75,866 


16,534 


29, 8< 


5,307 


0 


8,713 


136 9 2C1 


IV 


65,126 


20,377 


11,259 


4,925 


0 


13,961 


115,651 


V 


74,816 


33,420 


17,236 


11,449 


1,751 


42,473 


181,147 


Central 


44,105 


0 


2,105 


0 


0 


0 


46,210 


Total 


477,131 


151,020 


79,295 


36,550 


2,261 


111,568 


659,645 


Estimated Research 
Costs 


79,053 


0 


651 


0 


0 


0 


79,705 


Net Tot el 


$£,98,078 


$151,020 


$76,644 


$38,550 


$2,261 


$111,566 


760,140 



SOURCE: KDRC calculations froi/i WEET, JTPA, and Title )j< expenditure records; TOPS Or.-the-Jot Treining 
contracts; information gethered in staff interviews; and WEFT trecking system dete. 

NOTES: The enu of the observation period was November 1965 for opereting costs, end March 1965 for 
support service costs. 

e 

JTPA costs are those incurred in addition to TOPS expanses covered under contrects with WEET, end 
do not include wage subsidy paynentst 
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C. Other Program Services 

In addition to the TOPS sequence, members of the experimental group 
participated in regular WSET components, as well as seli-initiat<~d or 
program- referred education or training at JTPA, local schools, or 
ccmmunity-based organizations. The control grot,? also used these services, 
indeed, as indicated in earlier chapters, controls in saae instances 
received services similar to the ones offered in TOPS, although seldom more 
than one, never all three, and not in the same prescribed sequence. 

The use of non-TOPS employment and training services by experimentals 
and controls was identified from automated WEET and JTPA records. For 
those services provided under JTPA funding, pertinent average cost 
information could easily je obtained from JTPA. However, determining the 
costs of the other employment and training services, as well as uie use and 
cost of non-TOPS support services by the control group, required a match of 
a subsample of controls to individual case records, which was done by TOPS 
staff ♦ This match identified all eoucation, training, and other employ- 
ment-related services, as well as all support services recorded in the case 
records; in most cases, the full costs of these services could be 
identified by WEET staff, bjt in sane Instances they had to be estimated. 

As noted in Chapte^ IV, the unit costs of program services similar to 
the TOPS components were lower than for the TOPS components themselves. 
This at least partly reflects the fact that group services available to 
controls were generally delivered to larger groups than in TOPS. In 
addition, staff follow-up and monitoring was much less intensive for these 
activities chan for TOPS. 

The verage operating cost per experimental for these non-TOPS 
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activities was $108, with an additional $26 for support services, 
allowances and wage subsidies. Participation rates in these activities 
were lew for the experimental group. Overall, 3 8 percent of oxperimantals 
participated in scwe non-TOPS activity. 

Over L*\£ of the control group participated in seme employment or 
training activity. Also, rone of the controls who did not participate 
actively in a program component still received staff attention and/or 
support services, which entailed resource use that is Included in th±s 
analysis. The average operating cost of the services per control was $287, 
and the average cost of support services and subsidies was $230. The latter 
cost includes wage subsidies paid to controls in non-TOPS OJT as ignments 
(AFDC grant diversion was not used in these assignments). The total cost 
of $517 per control represents a substantial use of program resources. 
This is an important finding in itself, for it indicates that this analysis 
compares TOPS to a substantial alternative treatment. 

D. Total Costs for the Resources Used 

Two measures of the total value of resources used by the research 
sample are of interest; One is the gross cost of &11 program services 
received by the experimental group. This is the sum of the TOPS-sequence 
operating costs, the TOPS support services and allowances costs, and the 
costs of all non-TOPS services experimentals received. The gross operating 
cost per experimental proved to be $1,721; the gross cost of support 
services and allowances was $654, and the cost of wage subsidies was $386. 
Thus, the total col, of all TOPS and non-TOPS services provided to 
experimentals was $2,761. 

N et costs reflect the incremental use of resources by the experimental 
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group beyond that of controls. Thus f net costs have been calculated by 
subtracting the estimated costs of services per control from the gross 
costs per experimental that were just presented. The estimated net 
operating cost per experimental is $1,434, while the net costs of support 
services and subsidies are $521 and $289, respectively. The total net cost 
is thus $2,244. 

These net cost estimates can be expressed in 1985 dollars — as were 
the final estimates of benefits ~ by adjusting for inflation and discount- 
ing to reflect that base year. The resulting operating cost estimate is 
$1,461 per experimental, with an additional $531 covering allowances and 
support services and $254 for wage subsidies, which yields a total ret cost 
of $2,286. These estimates are the pertinent values for computing the 
overall benefit-cost results in the final section of this chapter. 

IV. Results 

The results for each type of program effect and resource use discussed 
in the previous sections are aggregated in Tables 6.8, 6.9, and 6.10. An 
individual result is entered as a benefit or cost for each perspective — 
depending on whether it represents a gain or a loss to that group. The 
results are then added together to estimate the net value of TOPS from each 
point of view. As indicated earlier, all estimates for society as a whole 
constitute the sum of the results for the welfare recipient and taxpayer 
perspectives; the latter includes the standpoints of both government 
budgets and employers. All results cover a five-year time period and are 
expressed in 1985 dollars. 

The first Arable presents the benefit-cost results from the perspective 
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•"ABLE 6.B 
MAINE 



FROM THE PERSPECTIVE OF THE WELFARE SAMPLE: 
ESTIMATEO GAINS ANO LOSSES PER EXPERIMENTAL AFTER FIVE YEARS 



Component of Ane Ly s i s 


Es t i ma t e 


Gains 




Ee rnings end Fringe Benefits 
On-the-job Treining 
Un subsidized Employment 

Allowances end Support Services 

AFOC Payments 


$ 652 
2642 
650 
472 


Losses 




Tax Payments 

Non-AFDC Transfer Peyments 
Use of Other Employment end Training 
Programs 


-753 
-562 

-119 


Net Present Velui* 6 

1 


3182 



SOURCE: See Tobies 6.4 end 6.7. 

NOTES: Results ere expressed in 1385 dollars. The full semple 
includes 297 expe ri men t els end 147 controls. Because of roundi--, deteil 
mey not sum to totels. Results include estimetes of projected h .ogram 
effects beyond the observation pt/riod [See Teble 6.4). 

e 

The net present value is tha sum of ell geins enc iobs'.s. 
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TABLE 6.9 



MAINE 



FROM THE GOVERNMENT BUOGET PERSPECTIVE 
ESTIMATED 6AIf;S ANO LOSSES PER EXPERIMENTAL AFTER FIVE YEARS 



Component of Analysis 


E 6 1 i me t e 




Gains 






Payroll Taxes 


$ 484 




Income and Seles Tex 


533 




Non-AFOC Trensfer Peyments 


562 




Trensfer Administration 


50 




U6e of Other Employment end 






Treining Pros rams 


302 




Lo 6666 






AFOC Peyments 


-4~2 




On-the-Job Training Wege Subsidy 


-294 




WEJT Opereting Cost6 


-1366 




Other Operating Co6ts 


-276 




Support Service Peyments 


-650 




Net Present Velue Q 


-1129 



SOURCE : See Tebles b.4 end 6.7 

NOTES : ReeuLte ere expre66ed in 1985 doLlers. The full senp.le 
includes 297 e xpe r i men t e 1 6 end 147 controls, Beceuse of rounding, deteil 
oey not 6um to totals. Results include estimates of projected progrem 
effects beyond the observation period (see Teble 6.4). 

e 

The net present velue is the sum of ell gains end losses. 
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TABLE 6.10 



MAINE 



EST I MATEO BENEFITS AND COSTS PER EXPERIMENTAL AFTER FIVE YEARS, 
BY RESEARCH GROUP AK) ACCOUNTING PERSPECTIVE 





Accounting Perspectives 


Component of Analysis 


Welfare Semple 


Budget 


Taxpeyer 


Society 


Earnings 
On-tha^Job Training 
ui^ubsidized Employment 


5BB 
2535 


-294 
0 


-588 
-2535 


0 
0 


Fringe Benefits 
On-the-Job Treining 
Uneubsidized Employment 


64 
307 


0 
0 


-64 
-307 


0 
0 


Output Produced by Participents 
Work Exper'encp 
On-the-Job Treining 
Uueubsidized Employment 


0 
n 

u 

0 


0 

n 
u 

0 


680 

ODD 

3056 


680 
665 
3056 


Tex Peymants 
Payroll Taxes 
Income end Sales Texes 


-220 


4B4 

533 


220 
533 


0 
0 


Trensfer Programs 
AFDC Payments 

Payments from Other Programs 
ArOC Administrative Costs 
Administrative Costs c, 
Other Programs 


-562 
0 

0 


562 
-26 

77 


— 4/c 
562 
- 26 

77 


U 
0 

- 26 

77 


TOPS Operating Costs 
WEET 

JTPA and Other Agencies 

Grent Divereion Administration 6 


n 
w 

0 

0 


— 1 ORG 
I ODD 

- 278 


— loob 
- 278 


-1366 
- 278 


TOPS Allowances end 
Support Services 


650 


- 650 


- 650 


0 


Use of Other Employment 
and Training ProgrBras 

WEET 

Other 


-119 
0 C 


254 
47 


254 
47 


136 
47 


Net Supervision Costs 


0 


0 






Preference for Work Over 
Welfare 0 


+ 


0 


+ 


+ 


Foregone Personal and Fami Ly 
Activities 




0 


0 




Net Present Value b 


31 62 


-1129 


• '2 


2990 



SOURCES: HDRC celculatlons from Unemployment InaurBnce records; AFDC dete; worksite eurvey; WEET 
Information Systea enrollment date; WEET and JTPA program cost recorde; TOPS On-the-Job Treining 
contrects; end published data on welfare coats, tex reteu, end employee fringe benefits. See text for 
deecriptions of these sources, 

NOTES.* Poeitive amounts Indicate a gain; nagetivo amounts indicate e Loes. ALL benefits end 
coets ere estimated for a fiva-yaBr time horizon beginning et eppUcetlon, end ere expressed in 19S5 
dollars. Because of rounding, deteil may not sum to totals. Results Include estimetes of projected 
progrem effects beyond .the observation period [eee Table 6.4). 

a ThBse era Intangible effects not measured in this enelysis. 

b The net preeent value 1s the eum of all gaina and Losses. 

c Eeti»atad velue loss then $0.50 end greeier then -$0.50. 




of w^xfare recipients, it should be remembered that these results reflect 
net gains and losses to recipients. For example, the gain to recipients 
for QJT earnings reflects the difference in such earnings between 
experimentals and controls (the latter group was, in a few instances, 
assigned to non-TOPS QJT positions). Similarly, the net loss associated 
with non-TOPS training programs is due to the fact that controls received 
aore allowances and support services from these programs than did 
experimentals. Hence, welfare recipients gave up tnis assistance by 
enrolling in TOPS. 

As the table shows, the group of recipients targeted by TOPS benefited 
on average by $3,182 per experimental over the five-year period. This 
reflects their steady gains in earnings and fringe benefits — from both 
OJT and regular jobs — which were offset somewhat by taxes, but not by 
reductions in transfers. Indeed, estimated AFDC and Medicaid payments 
actually increased; Food Stamps and unemployment compensation fell. 

Government budgets in contrast, shewed a net loss from TOPS of $1,129 
per experimental. The net cost of the program per experimental was about 
$2,300, easily the highest net cost among the welfare employment programs 
MDXZ has evaluated since 1981. Moreover, this reflects the relatively 
intensive use of education and training services by the controls, which 
partially offset the gross TOPS cost — including operating expenditures, 
allowances and support services and wage subsidies — of more than $2,700 
per experimental* The program generated a substantial increase in tax 
payments because of its large earnings impact, but no net savings in 
transfer payments. 

Table 6.10 presents the final benefit-cost results from the four major 
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perspectives considered in this analysis. It thus presents overall find- 
ings for taxpayers and for society in general as well as the budget and 
welt^re-recipient findings just described. The results indicate that 
taxpayers incurred a loss of about $200 per experimental. This is smtllor 
than the purely budgetary loss reported above due to e estimated net 
value of the OJT assignments to private employers and the value of the work 
experience services to government and nonprofit agencies. 

Given that the gains to welfare recipients and employers far exceeded 
the budgetary losses generated by TOPS, the net value of the program to 
society as a whole was substantial — almost $3,000 per experimental. This 
result clearly indicates that TOPS was, in overall economic terms, an 
efficient program. 

These Its are quite robust compared to other analyses of this 
kind. This is indicated by an assessment of the sensitivity of the results 
to assumptions used in the analysis, which entailed three types of 
sensitivity tests. First, tne net present value of TOPS was t;V *?ated for 
the four major perspectives using different assumptions in **ectl^g 
program effects beyond the observation period. Twelve alv tive 
assumptions were tried, four of which are shown in Table 6. 11. 12 Th* c 
shows how net present values change when any one of the projeoi. 
assumptions is altered in any of these four ways (keeping all other 
assumptions unchanged). Only one of the tests changes any of the 
qualitative conclusions of the analysis — that is, changes values from 
positive to negative, or vice versa — although other tests do alter the 
magnitude of seme values substantially. Specifically, the net value of 
TOPS to taxpayers becomes positive if it is assumed that the base period 



TA8LE 6.11 



NET PRESENT VALUE ESTIHATES 
GIVEN ALTERNATIVE ASSUMPTIONS, 
BY ACCOUNTING PERSPECTIVE 



Estimet -> Assumnti one 




u u u ll l» i ll y 


rcropBCt 1 V B 




We I f e re 

Sen p I e 


Budge t 


Texp ey e r 


Sod e ty 


Benchmark Estimate 0 


$3182 


-$1129 


-$192 


$2990 


Projection Assumptions 










No Post-Obse rve t i on Effects 


2237 


-186 8 


mm QQ A 

SOU 


1 O 13/ 


Desey Rete = 22 Percent 


3004 


-1287 


- 350 


2654 


8es«3 = Weighted Average of 










Last 5 Que r te r s 


3184 


-1163 


- 226 


2958 


8ese 50 Percent Higher 0 


3654 


- 760 


178 


3 83 2 


Upper-Bound Assumptions 0 


3816 


- 428 


510 


3395 


-ower-Bound Assumptions 0 


1753 


-2340 


-1403 


1280 


Output Assurap 1 1 on 8 










Value of Work Experience 










Output a zero 


3162 


-1129 


-872 


2310 


l/i sp lecemen t in OJT end 










Rcguter Employment = 










50 Percent 


31S2 


-1505 


-2052 


1130 



SOURCE i See Teble 6.10. 

NOTES: 8snchmerk estimates ere the net present value estimetes in Tebla 6.10. 
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estimate*, of earnings impacts (which are critical to the projected benefit 
estimates for taxes and Food Stamps) should be increased by 50 percent. 
This alternative assunption responds to the possibility of either 
measurement error or growth in impacts (rather than decay) during the 
projection period. 

Second, combinations of projection assumptions w~re tested. The 
results for ^he combinations that yielded the highest and xowest net 
present value estimates from each perspective are presented in Table 6.11. 
In this case one qualitative conclusion changed — the net present value of 
TOPS to taxpayers became positive — when the most favorable combination of 
assunptions vere used. It may be noted that while several changes in 
assumptions are involved in each combination, the critical factor is the 
projection of earnings. Fran the standpoint of welfare recipients, 
assuming zero future earnings impacts is key to the lower-bound estimate, 
and assuming a 50 percent highex base period earnings effect is key to t*<e 
upper-bound estimate. The same is true for taxpayers because higher 
earnings generate additional taxes and reduced Food Stamps. 

Finally, two differed,: employment assumptions were tested. In one 
test it was assumed 'jhat the net value of output from work experience 
positions was zero. in the other it was resumed that half of all regular 
and QJT employment resulted in th^ displacement of other workers — that 
is, the hiring of a TOPS participant meant \ot only that another person did 
not get the job, but also that he or she did not obtain other employment 
(and immediately replaced the TOPS participant on the welfare rolls). 
Neither change in assumptions altered the conclusions, although each change 
did affect the magnitudes of the net present value stiraattj. It is 
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especially worth noting the displacement assumption 1 s importance to 
taxpayers, since by definition they include the displaced workers who would 
lose employment. 

V. Conclusions 

The TOPS program generated a substantial redistribution of income to 
the welfare population it sought to help. The average gain in net income 
— taking into account earnings, taxes and transfers — was over $3,000 per 
experimental. Given that the net budgetary cost of the program was about 
$1,100, TOPS can be viewed as a much more efficient means of transferring 
income to this group than simply raising their AFDC benefits. 

Or. the other hand, TOPS was an expensive program that, despite its 
consistent impact on the earnings of welfare recipients, did not achieve 
any welfare savings. Thus, the conclusion reached about TOPS based on the 
results of this benefit-cost analysis ultimately rests on value judgments 
about the goals of programs like TOPS and the desirability of providing 
substantial help to the part of the welfare population targeted in this 
instance. 




CHAPTER VII 
POLICY IMPLICATIONS 

This chapter seeks to integrate several themes raised in earlier 
chapters and returns to the policy issues raised in Chapter I. In parti- 
cular, the discussion focuses on the significance of the TOPS demonstration 
within the larger framework of research on welfare employment programs 
conducted by MDRC and examines whether TOPS fulfilled the goals of its 
planners. Finally, the chapter considers seme of the implementation and 
management lessons that have emerged from the evaluation, and the 
conditions under which it would make sense to replicate the TOPS program 
model. 

Lessons on Comparing Program Effectiveness Across States 
In addition to the TOPS proqram in Maine, MDRC has completed evalu- 
ations of welfare employment initiatives in six states since 1981. In 
contrast to the voluntary and relatively expensive TOPS program, the other 
state initiatives were mandatory and lower-cost. Thus, there is a tendency 
to compare the impact findings from TOPS with the findings from other 
states to judge the relative effectiveness of alternative welfare employ- 
ment approaches. There are several important reasons, however, to be 
extremely cautious in making comparisons of this sort. These reasons 
underscore a more general lesson regarding the difficulty of comparing the 
effectiveness of different states 1 programs, given the diversity among 
states on such key dimensions as the welfare population served, APDC 



b^nef it structure and econcraic conditions. 

First, is the matter of program scale. In other states, MDRC evalu- 
ated a large part of the WIN or WIN Demonstration system, analogous to the 
full WEET system in Kaine. TOPS, hcwever, was only one small demonstration 
program within that system. A total of 297 individuals were enrolled 
statewide in TOPS over 15 months. This compares with an APDC caseload of 
16,556 in January 1983, and a WEET caseload of 3,157 registrants shortly 
before the start of th i demonstration. As discussed below, it seems 
doubtful whether the TOPS sequence could be run for a large proportion of 
the WIN population. 

A second, related distinction concerns the segment of the AFDC case- 
load involved in the programs. In most of the other states, one of the 
goals was to impose a participation requirement on a large segment of the 
WIN-mandatory population, comprised primarily of women whose youngest child 
was at least age six. Although some of the programs focused on a specific 
portion of the WIN-mandatory population (e.g., applicants or newly- 
mandatory recipients), few if any individuals within the eligible 
population were screened out of the program for lack of interest or 
suitability. Those who failed to participate without good cause could be 
sanctioned. In contrast, all TOPS participants had to apply to the program 
and go through a screening process before being enrolled. Staff were 
disinclined to sanction individuals for non-canpliance. In addition, as 
indicated in Chapter II, 61 percent of the TOPS enrollees were exempt from 
WIN participation, in most cases because they had young children. The TOPS 
research sample — which has many of the characteristics of a displaced 
hcmeraaker population — is thus not likely to be caoparable to the broader 
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range o£ individuals in the WIN-mandatory population. 

A third cautionary note concerns the features of the Maine welfare 
system* For example, the welfare caseload is overwhelmingly white, 
reflecting the state's population as a whole. Other states evaulated by 
MBRC generally have had a much higher minority representation on their 
welfare rolls. In addition, as was discussed in Chapter IV, the method of 
calculating AFDC grants in Maine creates greater work incentives than exist 
in most other states. This, together with the fact that Maine's economy 
improved during the period of the study, affects the expected mix of work 
and welfare within the AFDC population. 

Fourth, it must be stressed that in the mandatory programs evaluated 
by MDRC, the impacts on employment, earnings and welfare receipt, as well 
as the costs of the program, were averaged over a research sample of which 
approximately 50 percent did not participate in an activity. Non-partici- 
pants were appropriately included in the research sample because their 
behavior could have been affected by the program's mandate even though they 
did not participate (e.g., through deterrence or sanctioning), and because 
it is not possible to isolate the segment of the control group who were 
ccraparable to the experimental who did participate. In Maine, 90 percent 
of the experimental group received sane TOPS services. This higher 
participation rate it* presumably due in part to the fact that the program 
only served women who applied to participate and were screened for appropri- 
ateness. With non- participants comprising a smaller percentage of the 
research sample in Maine, the possible dilution of program impacts and 
costs caused by spreading them over non-participants as well as partici- 
pants is not as great as in other states. 
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A final factor to be considered in interpreting the results from the 
TOPS evaluation is the level of services received by individuals in the 
control group. As in all evaluations using random assignment, the impacts 
in this report are presented as the net difference between outcomes for 
experimental and controls, with the control group's outcomes potentially 
being affected by whatever services they may have received. Although 
members of the control groups in other states evaluated by MDRC were 
excluded from the special services being tested, they remained eligible for 
alternative employment-rela-ad services — for example, jtpa training, 
community college courses and, in some cases, limited WIN services. in 
Maine, the controls remained eligible for the entire range of WEET services 
and were excluded only from the full prescribed TOPS sequence, since the 
controls in Maine, like the experimental s, had demonstrated interest in 
receiving services and were screened to exclude those with serious barriers 
to employment, it is not surprising to see considerable activity in 
employment and employment-related services among this group. 

For all these reasons, caution should be applied in generalizing from 
the findings of the TOPS evaluation, or comparing these findings to the 
results from ot_or state programs. These qualifications should be kept in 
mind as the discussion now turns to some of the broader lessons that can 
properly be drawn from the Maine experience. 



II. Lessons on Targeting 

The targeting approach used in Maine illustrates a basic tension in 
welfare employment programs which seek to serve the harder-to- employ in 
components that are costly to operate and require lengthy periods cf 
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participation. On the one hand, staff want to provide such programs to the 
harder-to-es jloy because these individuals seem to need more intensive 
assistance. On the other hand, there is a natural tendency to select 
candidates who sear most likely to succeed, out of concern that expensive 
treatments will be wasted on those who drop out early or cannot become 
employable within the allotted timespan. This tension is even more 
pronounced in OJT programs that rely heavily on employer satisfaction with 
agency referrals. in Maine, the tension resulted in a targeting strategy 
that reflected two somewhat contradictory objectives. 

Maine intended to provide intensive services to women who qualified as 
harder- to- employ by virtue of having limited work experience and a history 
of welfare dependency. However, additional eligibility criteria specified 
that the women volunteer for services, be screened for barriers to partici- 
pation such as child care, health and transportation problems, and be able 
to read at a level commensurate with the materials in the pre-vocational 
training. These criteria were introduced to ensure that enrollees would be 
likely to complete the lengthy and intensive sequence of activities, but 
they also moved the program away from the use of objective criteria as 
enrollment standards. This tendency was further strengthened when local 
office staff established their own informal selection criteria to screen 
women according to their educational levels, their ability to demonstrate 
motivation, and the match between their skills levels and job objectives. 
This development, which had not been foreseen by the planners of TOPS, grew 
out of staff concern that the women be able to meet employer expectations 
and not be set up for failure, and that the program show good placement 
rates into OJT positions. 
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The result was a research sample that, judged by such objective 
criteria as length of welfare receipt and prior work experience alone, 
would be labelled harder-to-employ. However, the women 1 s educational 
status, demonstrated motivation and ability to arrange child care and 
transportation, all suggested that these women would be more likely to 
participate in employment and training activities and to find work than 
others who met the objective criteria but who did not ccme forward or were 
screened out. Indeed, the rates of cumulative participation in both 
employment-related activities and jobs shown by the control group in Maine 
were higher than for welfare recipients in both the experimental and the 
control samples in other states evaluated by MDRC. 

The experience in Maine suggests what might happen if states choose — 
as seme recent legislative proposals on welfare reform would encourage them 
to do a targeting strategy that sets objective enrollment criteria 
regarding length of welfare receipt and prior employment but gives priority 
to volunteers. Such programs may enroll a group of relatively employable 
women among those with long spells of welfare receipt and without recent 
work experience. This is all the more likely if, as in Maine, program 
guidelines specify that enrollees must be motivated and free of barriers to 
participation; staff are given considerable discretion in selecting among 
applicants; and the program is subject to performance standards that 
" 'nhasize high placement rates. 

An open question that cannot be addressed in this study is whether the 
screening had any effect on the impacts of the TOPS program. Although the 
program did produce sustained impacts on earnings, it is not known whether 
the impacts would have been greater or smaller in the absence of such 
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screening. It is noteworthy in this regard that intensive employment 
programs targeted to harder-to-employ welfare recipients can have impacts 
as large as or larger than those found for TOPS, even in the absence of 
screening and even when experiment als show relatively low rates of employ- 
ment and earnings. 1 This is because impacts measure the change that has 
occurred, compared to what would have happened in the absence of the 
program tested. 

It should also be noted that the restrictive screening contributed to 
the high cost of operating TOPS and was a major reason why the program 
scale remained small. Thus, ualess it can be demonstrated that screening 
increased the impacts of TOPS, the targeting strategy employed in Maine 
ultimately raises questions about the efficient use of scarce resources. 

Ill* Lessons on Meeting Program Goals 
A. Enrollment Goals 

The TOPS program fell well below its enrollment goals, due in part to 
the difficulties encountered in the outreach process and to the screening 
policies implemented at the local office level. While sane of these 
factors were unique to the implementation conditions in Maine, the failure 
to reach the enrollment targets is consistent with the pattern found in the 
other grant diversion OJT programs recently studied by MDRC. 2 These 
programs, too, fell short of their enrollment goals. These examples 
suggest that the scale of a grant diversion funded OJT program is likely to 
remain small. This kind of program should thus be seen as an employment 
strategy for a small proportion of the AFDC caseload, rather than as a 
strategy that could realistically be used for a broad segment of the AFDC 
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caseload, 

B. E mployment and Earnings Goals 

The planners of TOPS identified a number of goals regarding the out- 
comes of the program on employment and earnings. They intended that the 
TOPS sequence would both increase snployment levels and move AFDC recipi- 
ents into ."better; jobs — jobs that would pay higher wages, provide more 
hours of work and more stability than those which program eligibles would 
have obtained in the absence of TOPS. 

As discussed in Chapter V, the TOPS sequence had a positive impact on 
both employment and earnings. A higher proportion of experimental than 
controls was employed during each quarter of the follow-up period, 
beginning with quarter 3. It should be noted that the first two quarters 
of follow-up show negative impacts on employment, because participants were 
active in the pre- vocational training and work experience components. This 
pattern is typical in interventions that require lengthy participation and 
do not focus on immediate placement: early impacts are expected to be 
negative, as people are in program activities rather than in employment. 

There is some evidence that TOPS may also have helped people to move 
into better jobs. For example, as indicated in Chapter V, most of the 
earnings gains of $1,745 over quarters 2 through 11 (a 31 percent increase) 
were attributed to the fact that those with jobs were earning more while 
employed, rather than the fact that experiroentals merely had higher rates 
of employment. For those employed, the higher earnings resulted fran seme 
combination of higher wages, more hours of work per week or more weeks of 
employment. The separate contribution of each of these factors cannot be 
identified, however. TOPS may also have resulted in increased employment 
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stability, indicated by the 11 percent increase in the number of quarters 
in which experimental worked compared with controls (4.7 versus 4.2). This 
increase was not statistically significant, however. 
C. Goals Concerning Welfare Receipt 

An assumption underlying TOPS was thfet higher earnings would result in 
reductions in welfare receipt and the proportion of individuals receiving 
welfare. However, neither of these goals was achieved. On average, there 
was no net reduction in welfare receipt, and there may have been a small 
net increase. There is no fully satisfactory explanation for this 
phenomenon. Plausible explanations, discussed more fully in Chapter V, 
include the effect of the income disregard features that are inherent in 
Maine 1 s method of calculating welfare grants? the fact that sane of the 
earnings gains could have been for experimentals who would have left the 
welfare rolls anyway? and the effect on experimentals of being taught how 
to maximize their AFDC eligibility and to be their own advocates. 

IV. Lessons on Using OJT as a Transitional Employment Strategy 

TOPS sought OJT positions that would pay, on average, $4.00 per hour, 
be full-time in the private sector, include non-traditional placements and 
offer opportunities for training and stable employment. The characteris- 
tics of the OJT jobs, as discussed in Chapter III, indicate that the TOPS 
OJT positions met some but not all of these criteria. 

On average, the starting wage in the OJT p>sitions slightly exceeded 
the goal of $4.00 per hour, although two-thirds of the jobs paid $4.00 or 
less per hour. Just over a quarter were public sector or non-profit 
placements, reflecting both rollover from work experience assignments and 
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the attractiveness of public sector jobs, which frequently offered better 
wages and fringe benefits than private employers. Only a small proportion 
were in non-traditional jobs. Typically, the OJT positions appeared to 
offer little opportunity for training beyond that associated with showing 
any new employee the ropes. 

An unusual aspect of the TOPS design was the decision to place 
enrollees in pre-employment activities prior to placement in an OJT 
position, staff believed this would increase the program's success rate by 
raising participants' confidence and skill levels, thus making participants 
more attractive to employers and increasing the likelihood that they would 
complete the OJT assignment and continue as unsubsidized workers. The 
first components could also increase participants' employability by serving 
as a screening process for employers, providing more of a guarantee that 
the participants had demonstrated basic employability skills. Indeed, 
staff felt that in practice this up-front job preparation was important and 
did contribute to the high rollover rates in the OJT component* 

The evaluation, however, cannot determine the relative contribution of 
the pre-employment components to the impact results, and therefore cannot 
answer whether the additional cost was warranted. MDRC's soon to-be- 
ccmpleted evaluation of a grant diversion OJT program in New tfsrsey, where 
the primary difference between the experimental and control treatments is 
eligibility for OJT, may provide a better measure of the separate effect of 
an OJT program. 

Other policy issues concern the role of the OJT subsidy in TOPS, A 
number of hypotheses have been advanced to explain the practice of 
subsidizing wages paid by employers in OJT, One argues that enployers are 
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being compensated for higher training costs and lowered productivity while 
the new employee is learning the skills needed to p?rforra the job. Another 
argues that the subsidy functions as a ;risk premium; providing an 
incentive for employers to hire individuals who may appear less likely !;o 
succeed cn the job. However, if neither of these justifications applies 
and the participant would have been hired even without a subsidy, thin the 
subsidy might be more of a windfall than a hiring incentive. 

The TOPS findings shea seme light on this debate. A combination o£ 
factors — especially that the OJT positions seemed to offer little train- 
ing beyond that given to any new employee, and that employers found the OJT 
employees to be as productive or almost as productive as other new 
employees — suggest that in TOPS the OJT subsidy was not compensation for 
training costs o'r .reduced productivity. it does not follow, however, that 
even if employers benefit financially, an OJT position is inappropriate. 
The added cost of such a program might be worthwhile if it accelerates Jhe 
speed with which AFDC recipients are placed in jobs, or leads to better 
jobs. 

V. Lessons on Grant Diversion 

Another element of TOPS was the use of AFDC grant diversion to pay 
part of the cost of the employer subsidy in the OJT phase of the sequence. 
It is worth stressing again that grant diversion refers only to the funding 
mechanism; it is not the program intervention being evaluated. 

In general, the grant diversion process functioned smoothly, but a few 
cautions emerge frcra Maine 1 s experience. First, it is important to have 
seed money available to cover the period before the diversion becomes 
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effective and any short-falls in the monthly cash flow* It is possible 
that the diverted amounts will not pay for as high a proportion of the 
subsidy as projected. Second, the grant diversion pool must be carefully 
monitored on a monthly basis* Third, the timing of the transfer of funds 
from the AFDC account into the wage pool is critical: the start of the 
grant diversion period should be delayed to correspond with the period that 
an individual's grant will first reflect the QJT earnings* Fourth, if it 
is necessary to supplement or combine an automated tracking system with a 
manual one, it is doubtful that the wage pool could be operated on a very 
large scale. 

One rationale for AFDC grant diversion is that welfare savings fore- 
gone in the short run and invested in programs will yield larger welfare 
savings in the long run. TOPS, however, did not produce welfare savings 
during the follow-up period and, from the budget perspective, did not pay 
foe itself. This raises seme questions regarding the returns on the grant 
diversion investment* However, because the component funded with grant 
diversion was only one piece of a multi-part sequence in TOPS, the separate 
effect of the grant diversion strategy on long-term AFDC expenditures 
cannot be isolated. It should also be kept in mind that from the social 
perspective, the benefits of TOPS exceeded the costs. 

VI* Other Management Lessons 

TOPS posed an implementation challenge because of the complexity of 
the multi-ccraponent design, the necessity for interagency coordination in 
service delivery, and the length of time that participants were expected to 
be active in the program. The demonstration experience suggests a number 
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of general policy lessons of relevance to other states. 
A. Lessons on Interagency Coordination 

The TOPS program illustrates sane of the difficulties inherent in 
efforts to coordinate the provision of training and placement services for 
APDC recipients between the WIN or WIN Demonstration system and the JTPA 
system. Despite a mutual commitment to serving AFDC recipients in TOPS, 
the two staffs in Maine displayed quite different philosophies about 
whether and how to work with harder- to- employ clients. They applied 
different criteria to assess whether individuals were appropriate for 
services and whether they were job ready; JTPA staff were particularly 
sceptical about the advisability of determining in advance who would be a 
good candidate for CXTT several months later. The two staffs also had 
different attitudes about what it was appropriate for staff to do for 
clients and what clients should be expected to do for themselves. In 
general, JTPA expected program enrollees to be more self-reliant and less 
dependent on support services and staff counseling and intervention. 

These differences reflected the very divergent standards by which 
performance was measured in WEET and JTPA, the particular resource 
constraints cn the two systems and the relationship each had with the 
employer community. JTPA staff were more accountable for — and thus, more 
concerned about — high placement rates, quick turnover, and low-cost 
initiatives. Seeing themselves as responsible to the employer community as 
well as to the disadvantaged, they were also more concerned than WEET about 
employer satisfaction. 

Additional tensions arose from the poor ccmmunication channels that 
were in place at the start of the demonstration, difficulties compounded by 



the fact that participants moved back and forth between agencies during 
successive components of the TOPS sequence. Communication problems were 
further exacerbated by the fact that the political relationship between the 
two systems was changing in Maine, and each was attempting to carve out a 
new role in providing employment and training services to disadvantaged 
groups . 

The philosophical differences between WEET and JTPA staff had two 
primary effects on the implementation of the demonstration. First, they 
led to the development of more stringent screening criteria than had been 
intended in the design. Secrnd, they led to seme confusion among partici- 
pants about the services to be provided, staff expectations and their own 
responsibilities. 

As discussed more fully in the interim report, Maine's experience 
suggests that the following items are important to the smooth implementa- 
tion of a program that relies heavily on interagency coordination and case 
management for service delivery in a multi-ccmponent sequence of activites: 
the involvement of representatives of local as well as central staff frcm 
both agencies in the early planning stages; the designation of a liaison in 
each local office who commands the respect of staff in both agencies; joint 
staff training at periodic intervals, beginning before the start of the 
program; regular opportunities for case-management reviews with staff from 
both agencies; and the need for continued attention frcm top level staff in 
both agencies. 

B» Lessons on the Fit between Management Style and Program Design 
The implementation of TOPS also provides ccme general lessons about 
what is likely to occur when a fairly prescriptive program model is 
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superimposed on a system in which the management structures encourage 
discretionary decisionmaking by staff. 

The WEET system was premised on the idea of discretionary decision- 
making both at the regional office level and at the case-manager level 
within each regional office. This was a conscious decision by central 
planners who wanted to avoid the over-regulation they associated with the 
WIN system that WEET replaced. To encourage regional diversity, staff in 
the Division of Welfare Employment developed flexible guidelines for the 
use of resources, program mix, service priorities and performance goals. 
At the WEET staff level, the concept of case management and service 
brokering required staff to tailor clients' employability development plans 
to their individual needs and to use resources in a flexible manner. 

The TOPS program model, in contrast, was quite prescriptive in a 
number of important ways. As conceived by planners in the Division of 
Welfare Employment, the intake process would have precluded much decision- 
making by staff, relying instead on the application of objective criteria 
to determine eligibility. The TOPS model, as designed, was also prescrip- 
tive in that it specified a sequence of activities through which partici- 
pants would move and did not provide for alternative service options once 
people had begun the sequence. Because this approach contrasted noticeably 
with service provision and case-management responsibilites in the rest of 
WEET, the restrictions were difficult for both clients and staff to accept 
and accounted for much of the staff's dislike for the demonstration. 

The different philosophies also resulted in seme adjustments to the 
design in practice. As previously discussed, WEET staff — influenced 
largely by JTPA staff — tended to do more screening than planned prior to 
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random assignment and before transitioning participants into subsequent 
components. They also added job search activities not called for in the 
program model. Finally, subset of the experimental group enrolled in 
education and training activities that were not part of the TOPS design. 

The point is not that prescriptive models should be avoided, but that 
it will be difficult to operate such designs within a management structure 
that pushes for greater freedom in decisionmaking. in these cases, it is 
likely that staff will find ways to adapt the model to conform more closely 
with traditional management practices, particularly if the new program is 
only a small piece of the overall system. 



VII. Lessons on Replicating TOPS 

Whether states should seek to replicate the TOPS program model is 
essentially a value judgment that cannot be answered on the strength of the 
research findings alone. TOPS was clearly effective, compared to the alter- 
native services used by the control group, in raising the income levels of 
the welfare recipients who volunteered to be in the program. However, from 
the budget standpoint, the approach was not cost-effective: the costs of 
operating the program were not offset by the additional taxes paid and any 
budgetary savings. 

If a state is primarily concerned about raising the incomes of AFDC 
recipients — in this case, those who are working — and less concerned 
about achieving reductions in AFDC benefit costs, TOPS could be an appeal- 
ing program to operate. Indeed, it appears to represent a rather efficient 
method of transferring income: a net budgetary cost of $1,129 per experi- 
mental resulted in a net income gain of $3,182 per experimental after five 
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years. 

The scale of the program is also an Important consideration. Because 
a program like TOPS will probably remain small, it is not likely to be a 
major component of a welfare employment strategy. It could, however, be a 
premising piece of such a strategy if implemented as one alternative among 
an array of services. 
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TABLE A.1 



MAINE 



SH.ECTEO CH/KACTERISTICS OF AFOC SAMR.E MEffiBIS AT THE TI« OF RE;E/«CH STAflT, 

BY DEMONSTRATION STATE 



Charactarl 8t1 c 


Maine 


Arkansas 


San Oiegoi 
Californie 


Cook 
County t 
Illnois 


Baltimore, 
Mery land 


Virgi nia 


West 
Virgi nie 


Study Area 


State- 
Wide 


Eight 
Counties 


County- 
Wide 


County- 
Wide 


10 of 18 
Centers 


11 Cities/ 
Counti es 


21 

Counti es 


Welfare Status (X) 
Appl leant 
Reel pi ent 


0.0 
100.0 


59.5 
40.5 


100.0 
0.0 


d 

0.0 
100.0 


49.8 

r-n o 
WW .C 


40.4 
59.8 


0.0 
100.0 


WIN Status (X] 
Mandatory 
Voluntary 


38.8 
81.4 


100.0 
0.0 


100.0 
0.0 


100.0 
0.0 


100.0 
0.0 


100.0 
0.0 


100.0 
0.0 


Age (X) 

24 Yeers or Lsss 

25 to 34 Yeers 
35 to 44 Yeers 
45 Years or More 


22.7 
4S.4 
22.9 
5.0 


26.8 
50.8 
15.9 
6.5 


9.1 
46.6 
33.7 
10.6 


25.7 
45.6 
20.5 
8.3 


14.0 
52.8 
24.7 
8.4 


8.6 
49.0 
31.4 
10.9 


4.9 
46.8 
36.1 
11.9 


Average Age (Yeers) 


30.1 


29.4 


33.6 


N/A 


31.9 


33.6 


34.6 


Sex (X] 
Male 
Feaale 


0.0 
100.0 


2.4 
97.6 


15.6 
84.4 


12.3 
87.7 


10.1 
89.9 


0.0 
100.0 


0.0 
100.0 


Ethnicity (X] 
White, Non-Hi spenic 
Blackt Non-Hi spenic 
Hi spenic 
Other 


97.9 
1.2 
0.5 
0.5 


13.4 
86.3 

0.1 B 


56.9 
20.5 
18.2 
4.5 


15.7 
72.1 
11.1 
1.1 


29.5 
89.? 
0.3 
1.0 


32.8 
64.0 
1.2 
2.0 


89.9 
9.8 

0.3 8 


Degree Received (X) 
None 

General Equivalency Diplaae 
High School Diplacs 


25.0 
24.8 
50.2 


50.4 
49. 6 f 


39.1 
7.5 
53.4 


N/A 
N/A 
N/A 


58.4 
6.8 
36.9 


56.3 
8.1 
35.5 


54.0 
15.2 
32.8 


Average Highest Grade 
Con pi eted 


11.3 


11.0 


11.3 


N/A 


10.5 


10.5 


10.2 


Marital Status (X] 
Never Married 

Marriedf Living With Spouse 
Married, Not Living With 
Spouse 

Divorced or Widowed 


25.5 
0.5 

13.6 
80.4 


48.9 
2.7 

25.1 
23,3 


16.0 
12.9 

34.1 
37.0 


N/A 
N/A 

N/A 
N/A 


40.5 
8.8 

33.5 
17.2 


31.1 
7.1 

35.7 
26.1 


13.3 
16.8 

22.8 
47.0 


Average Number of Children 
Under 19 Years 


1.81 


N/A 


1.74 


N/A 


1.80 


N/A 


2.00 


Any Children (X) & 
Lese Then 6 Years 
Between 8 and 18 Years 


49.3 
68.0 


53.7 
68.7 


1B.4 
87.5 


N/A 
N/A 


16.8 
86.5 


N/A 
N/A 


10.8 
94.6 
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TABLE A.1 (continued] 



Charscteri et1c 


Maine 


Arkansas 


San Diego, 
Cel1forn1e 


Cook 
County , 
Illinois 


8al timore, 
Hery lend 


Virginia 


Vest 
Virginia 


Prior AFDC Deptndency (X) 
never on Artiu 
Two Year 8 or Less 
More Than Two Years 


n n 

38.9 
63.1 


QC Q 
OD.9 

32.7 
30.5 


So*/ 
38.9 
27.4 


N/A 
N/A 
N/A 


13.9 
31.5 
54.7 


12.1 
28.1 
59.8 


13.9 
31.9 
54.2 


Average Months on AFDC During 
^o Yaara Prior to Raaoarch 
8 tart 


19.4 


11.0 


8.1 


N/A 


13.7 


14.3 


14.0 


neio a Job at Any Time During 
Four Quarters PMog to 
Raaearch Start (%] 


34.7 


22.5 


51.5 


31.1 


44.3 


36.7 


17.9 


naio c wod at Any Time During 
Quarter Prior to Raaearch 
Stert (%) 


16.2 


14.1 


33.1 


18.5 


28.2 


25.6 


9.6 


AvaraQe Months Eaployed 
During Two Years Prior to 
Research Start 


3.2 


5.2 


10.1 


N/A 


5.5 


N/A 


2.8 


Cnn 1 nnn» «+- Ink Ual A J m 

ror Longest uop nolo in 

Piet Two Year 8 
Average Hourly Wage Rate(l] 
Average Weekly Hours 
Duretlon of Job (Months] 


3.64 

on a 
oU.4 

13.0 


3.78 
oo.b 
19.3 


5.13 

nn n 

oo.o 
22.0 


N/A 
N/A 
N/A 


4.79 
33.1 
28.7 


3.87 
29.9 
17.9 


3.74 
33.7 
17.5 


Sea pi a Size 


444 


1153 


3591 


11812 


2623 


3184 


3694 


Seeple Enrollment Period 


10/83- 
1^B4 


6/83- 
a/84 


10/82- 
8/83 


2/86- 
9/86 


11/32- 
12/83 


8/83- 
9/84 


7/83- 
V84 



SOURCE* Calculations from MDRC Client Information Sheet6 as reported 1n HDRC Final Reports for 
Arkanaaaj San Diego. California* Illinois; Maryland; Virginia; and West V1 rg1 n1 a; Table 2.3 of this report. 

NOTES r The samples include both Experimantals and Controls. Ths table Includes only AFDC clients; 
however, some state demonstration programs include bo: s AFDC and AFDC-U cllsnts. 

N/A Indicates data not available. 

Distributions may not add exactly to 100.0 percent because of rounding. 
No tests of statistical significance were calculated. 

"Distributions may not add to 100. 0D percent because individuals can have children in more than 

one category. 

b 

Calculated from Unemployment Insurance earnings records. 

c 

For questions concerning longest Job, staple sizes are based on the number of people who report 
a longest Job on the Client Information Sheet or Client Aaaesamant Form. 

d 

Although all sempta members were receiving AFDC at the time of randoo assignment, 33.7 percent 
had not received AFDC in the quarter prior to random assignment, 
e 

Includes Hispenics. 
^Includes General Equivalency Diploma recipients. 
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APPENDIX B 

THE MECHANICS OF ADMINISTERING THE TOPS GRANT DIVERSION WAGE POOL 

I* The Grant Diversion Process 

In Maine, the diversion of the APDC grant involved staff from regional 
WEET offices, regional and central AFDC Income Maintenance Unit ( IMU) 
offices, and Department of Human Services (DHS) accounting staff ♦ 
Reimbursing employers for the wage subsidy involved both WEET and JTPA 
staff. 

A. Diverting AFDC Grants into the Wage Pool 

1# Notification of the Income Maintenance Unit 
Once an OJT contract was written for a TOPS client, the regional 
WEET office notified the regional Income Maintenance Unit (IMU) of the 
placement. The notification form included information abput the employer's 
name and address, the job start date, initial pay date, wage rate and the 
number of months the OJT was scheduled to last. The form was filled out 
manually by the WEET specialist and sent on to the IMU. After additional 
information was added by IMU, a copy was returned to WEET. 
2. Calculations by the IMU 

Calculating the amount of money to be diverted into the wage pool 
for each client was a two step process, done by the regional IMU case- 
workers. First, it was necessary to identify the period of time during 
which the funds would be diverted. Second, it was necessary to calculate 
hew the OJT earnings affected the AFDC grant level. 

To calculate the period during which the funds were to be diverted, 
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the IMU caseworker determined what was known as the /effective month of 
grant diversion opening; and the .'effective month of grant diversion 
ending.; These were distinct from the calendar months of the OJT start- 
and end-dates. Because Maine uses retrospective budgeting, there was a lag 
(generally of two months) between the point the client started working in 
an OJT position and the point when the grant level was affected by the new 
earnings. For example, income received in January would not normally 
affect an AFDC grant until March. 

For this reason, the imu did not begin diverting the grant as soon as 
the client started working in an OJT position, but waited until the first 
month the grant would be affected: this was ;the effective month of grant 
diversion opening.; Similarly, the effective grant diversion closing date 
was calculated f rem this effective grant diversion opening date rather than 
from the actual end-date of the OJT contract. Thus, if an OJT contract 
were written for a period of three months, the funds were diverted for 
three months after the effective start date, even though the client would 
have completed the OJT after the first of those three months. 

Next, the IMU caseworker calculated how the OJT earnings affected each 
client's grant. in TOPS, the base grant was ;frozen; at the start of the 
effective OJT period, but the IMU caseworker recalculated the monthly grant 
throughout the diversion period, using the same income disregards that 
normally apply to a working recipient. if the incane was high enough to 
make the person ineligible for AFDC, the client did not receive a welfare 
check. However, her AFDC case was not closed as she remained eligible for 
Medicaid throughout the grant diversion period. 

Recalculating the individuals grant on a monthly basis meant that the 




-173- 



215 



amount of money diverted into the pool each month fluctuated along with the 
fluctuations in the APDC grant* The fluctuations reflected the fact that 
earnings varied from week to week and month to month as the number of hours 
worked varied* Monthly child-care costs also varied* 

Once the effective date of the OJT period was determined, the IMU 
worker calculated the initial change in benefits and entered it on the 
notification form sent by WEET, along with the effective start and end of 
grant diversion and the base (frozen) grant amount* A copy of this form 
was returned to WEET. 

The effective month of che grant diversion opening and closing and the 
base grant at the start of the grant diversion period were also entered 
into the computerized APDC files* This created an automatic trigger to 
start and end the effective grant diversion period, and a way of keeping 
track of the amount of money being diverted each month* 

If a client terminated her OJT prior to the expected completion date, 
the WEET specialist notified the IMU worker, identified the reason for the 
early completion, and indicated the new effective month of grant diversion 
closure* All of this information was added to the original notification 
form filled out by WEET* 

3 ♦ Transferring Funds into the Wage pool 

The dollar amount diverted into the wage pool was the difference 
between the client's base grant at the effective start of the OJT period 
and the actual grant she received during each month of the effective OJT 
period* This calculation was computerized, based on the information in the 
APDC files* 

Each month, a print-out containing the name oi each TOPS client in an 
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OJT position in the state, her base grant, her currrent grant, and the 
amount to be diverted into the OJT wage pool was generated from the 
automated AFDC files. The print-out also computed the total amount to be 
diverted throughout the state. Copies were sent to the central WEET 
office, the central IMU office, and the DHS accounting office. 

Based on these figures, the DHS accounting office proces^d the actual 
transfer of funds from the AFDC account to the grant diversion pool, a 
separate account. This was simply a bookkeeping process. It was sometimes 
necessary v , however, for the accountant to do a manual adjustment to the 
diversion calculation. It was possible, for example, that the amounts to 
be diverted for individual clients would show up as negative figures. 
These had to be converted to zeroes and the totals recalculated before the 
funds were transferred into the wage pool account. 

B. Using the Wage Pool to Reimburse Employers 

The cost " the employer subsidy in TOPS was shared by JTPA and WEET, 
with WEE" *vv„ ^ng 70 percent and JTPA, 30 percent., However, to simplify 
the reimbursement process, it was agreed that when JTPA wrote an OJT con- 
tract, JTPA would pay the employer the full subsidy, and then bill WEET for 
70 percent of that cost. Considerable lag time developed as many employers 
did not begin to bill JTPA until three or four months after the start of 
the OJT position. (They were supplied witii a monthly invoice form from 
JTPA.) Because of this lag, local JTPA offices did not bill WEET on a 
monthly basis, but at 45- or 90-day intervals. 

After the JTPA invoice was received, the accountant in the WEET 
regional office instructed the regional DHS office to pay the bill, using a 
general WEET acc . rather than the special grant diversion wage pool 
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account* In another bookkeeping transaction, an accountant in the -*ntral 
DHS office arranged to transfer funds out of the wage pool account into the 
general WEET account to cover the payments for the employer subsidy. This 
was done on a monthly basis, after ceatral DHS accounting received copies 
of the invoices paid by each regional DHS office to JTPA. 

II. Implementation Issues 

Officials in Maine developed a grant diversion process that reflected 
some of the unique aspects of welfare payments in Maine and which other 
states interested in using grant diversion need not duplicate. These 
decisions and their implications for the smooth implementation of the grant 
diversion wage pool and employer reimbursement warrant discussion. 

• Although Maine froze the base grants on which the diversion 
amount was calculated for the duration of the OJT period, the 
actual grant was recalculated monthly to reflect changes in 
the client's income and income disregards. 

This was done because the IMU staff felt it was important to treat the 

TOPS clients, as much as possible, .'like everyone else* on AFDC. In terms 

of the AFDC grant calculation process, the only thing which distinguished 

the TOPS OJT participants is that their cases were not closed and they 

therefore continued to receive Medicaid — if their income level made them 

ineligible for AFDC. 

• In order to parallel the lag time inherent in retrospective 
budgeting for AFDC grant calculations, the grant diversion 
pc-riod was based on an effective OJT opening and closing date. 

Prior to the enactment of this procedure, the grant diversion period 

was calculated from the start of the OJT job rather than from the date when 

the earnings first affected the grant level. As a result, an insufficient 




amount of funds was intially being diverted into the wage pool. 

• Because the IWU and WEET computer systems were not compatible, 
the linkages between the two systems had to be made manually, 
by physically sending notification of the status of individual 
clients back and forth between offices. 

This increased the likelihood that an individual in OJT might not be 

entered into the APDC files or that if the OJT terminated earlier than 

scheduled, the monthly diversion system would not be halted. Both did 

occur in the initial months of grant diversion in TOPS. 

• The original computer program used by the IMU staff to 
calculate the amounts to be diverted to the wage pool did not 
include the end date of the diversion period. 

As a result, in sane cases funds continued to be diverted for months 

after an individual left her OJT position. To remedy this, staff first 

programmed six months — the maximum length of OJT — as an end date, and 

subsequently included the actual end date of the effective grant diversion 

period. 

• The WEET regional staff did not use a special billing code to 
reimburse JTPA for WEET's share of the OJT subsidy, but 
instead drew the funds out of the general WEET account. 

This necessitated an additional transaction at the central accounting 
office, because the total amount paid to JTPA in a given month had to be 
transferred from the wage pool account into the general WEET account. 

For all these reasons, there were seme problems in the administration 
of the grant diversion mechanism during the early phase of its administra- 
tion. At one point, the amount needed to cover the employer payments 
exceeded what as available in the pool, and regular WEET funds had to be 
used to cover the difference. A review of the grant diversion records 
identified funds which should have been diverted, but had not been. New 
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procedures, including the use of the effective start- and end-dates of 
grant diversion, programming the end-date of grant diversion into the AFDC 
system, more intensive monitoring of the computer records, and more stafi 
training, resulted in a smoother- running system. The process continued to 
require close monitoring and an occasional manual intervention, however. 

The experience of managing grant diversion in TOPS led WEET staff to 
make the following recanmendations for the implementation of a grant 
diversion wage pool in other states: first, the mixture of automated and 
manual procedures would probably not work well with a much larger-scale 
grant diversion program; instead, it would be necessary to integrate the 
WEET and AFDC files. Second, because of the lag time caused by the 
retrospective AFDC budgeting and the employer's slowness iii Dilling JTPA, 
it is difficult to predict whether the pool will always have sufficient 
funds, or whether it is paying for itself at any point in time. For the 
same reasons, a grant diversion system would probably require a supple- 
mental source of funds to get it started. 
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TABLE C.1 



MAINE 

ESTIMATEO REGRESSION COEFFICIENTS FOR SELECTED MEASURES 
OF EMPLOYMENT ANO AFDC RECEIPT 





Oependent Variables 


Rnnrnfifin r 


Ever Employed, 
Quarters 2-11 


Averege 
Tote I Eerning&g 
Querters 2-11 [S] 


Ever Received 
AFDC, 

e 

Querters 2-11 


Averege Totel 
AFDC Income, 
Querters 2-11 [$] 


Constant 


0.802*** 
(0.032) 


5599.172*** 
[638.884] 


0.980*** 
(0.011) 


6598.520*** 
(313.270) 


Experimentel 
Status 


0.016 
[0.039] 


1744.830** 
(786.748) 


0.003 
(0.014) 


169.934 
(385.774) 


Region 
I 










II 


0 t 022 
[0.061] 


-2265.256* 
[1224.002] 


0.006 
(0.021) 


507.587 
[600.176] 


III 


0.077 
[0.059] 


-2368.146** 
(1182.571) 


0.005 
(0.021) 


428.532 
(579.861) 


IV 


-0.069 
[0.056] 


-3287.434*** 
[1128.633] 


-0.007 
(0.020) 


368.117 
(553.413) 


V 


0.014 
[0.054] 


-2876.745*** 
(1086.132) 


0.003 
(0.019) 


296.050 
(532.573) 


Age 

24 Years Or Less 


0.051 
(0.057) 


-234.597 
[1144.093] 


0.028 
(0.020) 


341.206 
(560*994) 


25 to 29 Years 










30 Yeers Or More 


0.019 
(0.045) 


364.270 
[914.945] 


0.008 
(0.016) 


-266.946 
(448.633) 


No High School 
Oegree 


-0.157*** 
[0.043] 


-3103.450*** 
(871.691) 


-0.016 
(0.015) 


586.816 
(427.424) 


Never Married 


0.112** 
[0.051] 


644.148 
(1018.587) 


-0.002 
(0,018) 


-2.832 
(439.453) 
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TABLE C.1 (continued] 





Dependent Veriebles 


Regressor 


Ever Employed* 
Querters 2-11 


Ann nann 

Average 
Totel Earnings, 
Querters 2-11 [$] 


Ever Received 
AFDC, 

e 

Querters 2-11 


Averege Totel 
AFDC Income, 
Querters 2-11 [$] 


Number of Children 
One 


-0.085* 
(0.044) 


726.416 
(879.539) 


-0.021 
(0.015] 


-1983.941*** 
(431.272) 


Two or More 


-0.002 

f n rM c 1 


138.793 
1 297. 279 J 


-0.002 
(0.005) 


643.572*** 
(145.767) 


Received AFDC for 
More then Eight 
Quarters Prior to 
Random Assignment 


0.023 

f n n a 1 1 
10.041 J 


-536.324 

t O Q ftp 1 

1823.825] 


-0.028* 
(0.014) 


498.731 
(403.954) 


Held Job Anytime 
During Four Quarters 
Prior to Random 
Assignment 


0.163*** 
(0.045] 


-142.042 
(912.092] 


0.D30* 
(0.016] 


-131.413 
(447.234) 


Earnings Greater then 
$1500 During Four 
Querters Prior to 

Rflnrfom Arqi nnmflnt' 


-0.061 
I U. Ubb J 


2884.282** 
11323.320] 


-0.009 
(0.023] 


-307.377 
(648.876) 


Win Mandetory 


D.012 

fn rMCl 
I U.U4oJ 


860.482 
IB99.B92 J 


0.000 
(0.016] 


-984.733** 
(441.253) 


Number of 
Observations 


444 


444 


444 


444 


Number of 
Experimentels 


297 


297 


297 


297 


Number of Controls 


147 


147 


147 


147 


Degrees of Freedom 
For Error 


428 


428 


428 


428 


Error Meen Squere 


0.143 


58257821.871 


0.018 


14007079.717 


R-Squere 


0.090 


0.084 


0.032 


0.178 


Meen of Dependent 
Verieble 


0.813 


6766.22 


0.982 


6712.191 



(continued ) 
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TABLE C.1 (continued] 



SOURCE: HDRC calculations from State of Maine AFOC end Unemployment Insurance earnings 
records, 

NOTES: Ordinary leest squares regression coefficients in this teble correspond to itnpect 
estimates presented in Tebles 5.2 r 5.3 f 5.9 r end 5.10. An analysis of coverience procedure wes 
used to control for 14 kinds of differences in characteristics befora random assignment. See 
Ostla (1975 r p. 461). Standard errors of coefficients ere indicated with parentheses. 

These calculetions include sample members not employed end sample members not 
receiving AFOC. 

A two-tailed t-tast was applied to eech coefficent astiraete. Stetisticel 
significance levels ere indiceted es: *=10 percent; **=5 percent; ***=1 percent, 

a 

Quarter 1 r the querter of rendotn essignment, mey contein some income from the 
period prior to random assignment and is therefore excluded from cumulative meesures of 
employment eernings r end AFOC receipt. 
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TABLE C.2 



MAINE 

IMPACTS OF THE TOPS PROGRAM ON THE JOINT 0ISTRI8UTI0N 



OP EARNINGS AN 0 


APOC INCOME, 


QUARTERS 1-11 






Outcoae and FoLLow-Up Period 


Experlaentel a 


Control e 


01 f f erence 


P 


Eaployaent and Welfare Statue, 
Quarter 1 [%} 










Hed No Earnings, Received No APOC 
Payaente 


1.4 


0.6 


0.7 


0.498 


Hed Soae Earnings, Received No APOC 
Peyaent e 


0.0 


0.0 


0.0 


— 


Had No Eernlnge, Received Son APOC 
Payaente 


82.2 


74.7 


7.5» 


0.060 


Hed Son Eernlnge, Received Soae 
APOC Payaente 


16.4 


24.7 


-8.2«* 


0.033 


Total 


100.0 


1 00. Q 


0.0 




Eaployaent and Welfare Status, 
Quarter 2 [%] 










Had No Eernlnge, Received No APOC 
Payaente 


1.7 


0.6 


1 .2 


0.325 


Hed Soae Eernl nge, Racelved No APOC 
Peyaent e 


0.6 


1 .4 


-0.8 


0.420 


Hed No Earnings, Received Soae APOC 
Payaente 


74.8 


64.6 


10.2" 


0.026 


Hed Soae Earnings, Received Soae 
APOC Payments 


22.9 


33.4 


-10.6«« 


0.018 


Totel 


100.0 


1 00.0 


0.0 




Eaployaent end Welfere Stctus, 
Quarter 3 [%] 










Hed No Eernlnge, Racelved No APOC 
Payaente 


3.3 


5.0 


-1.7 


0.388 


Hed Soae Eornl nga, Received No APOC 
Peyaent a 


6.9 


D .? 


1 .7 


0.503 


Hed No Earnlnga, Received Soae APOC 
Peyaante 


53.9 


55.7 


-1 .8 


0.723 


Hed Soae Eernl nge, Racelved Soae 
APOC Payaente 


36.0 


34.1 


1 .8 


0.707 


Totel 


100.0 


1 00.0 


0.0 





( con tl n uod ) 
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TA6LE C.2 (conti nued] 



Outcoaa end FoLLow-Up Period 


CXpB " I IBilf B I B 


Pn nt rnl • 
uu II h r u I ■ 


n i f f iPinn 

Ull 1 II III 




Eaployaent and Welfare Stetue, 
Quarter 4 [%} 










Hed No Eernlnge, Received No AFOC 
Peyaent e 


^ n 
0 • u 


*♦ • a 


0 .0 


0 .3 65 


Hed Soae Eernlnge, Received No AFOC 
Payments 


i a • d 


in n 


o • o 




Hed No Eernlnge, Received 9oee AFOC 
Peyaente 


43.0 


51 .3 


-6.3 


0.109 


Hed Soee Earnlnga, Received Soae 
AFOC Payaente 


36.6 


33.6 


5.0 


0.310 


Totel 


i nn n 
1 u u • u 


100.0 


0.0 




Eaployaent and Welfere Statue, 
Quarter S l*J 










Hed No Earnings* Received No AFOC 
Pay is nt e 


7 .5 


6.7 


-1 .2 


0.662 


Hed Soee Eernl nge, Received No AFOC 
Peyaents 


lOiC 


14.2 


1.1 


0 .767 


Had No Earnings, Received Soae AFOC 
Peyaents 


42.6 


50.0 


-7.4 


0.149 


Had Soae Earnings, Received Soae 
AFOC Peyaents 


34.7 


27.2 


7.5 


0.106 


Total 


inn n 

1 UU |U 


100.0 


0.0 




Eaployaant and Welfare Status, 
Querter 6 {%} 










Hed No Eernlngs, Received No AFOC 
Pay a ante 


6.3 


6.6 


-3.5 


0.200 


Hed Soae Earnlnga, Received No AFOC 
Payne nt a 


19.9 


16.9 


1 .0 


0.794 


Had No Earnlnga, Received Soae AFOC 
Pay aenta 


45.8 


49.5 


-3.7 


0.471 


Hed Soae Eernlngs, Received Sobs 
AFOC Peymente 


27. S 


21 .6 


6.1 


0.162 


Totel 


100.0 


100.0 


0.0 





( conti nued] 
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TABLE C.2 (connl nued) 



Out coat and Foil our- Up Period 


Expe Mb ntal a 


Control a 


01 f f er ence 


P 


Eaployaant andWalfara Statue? 
Quarter 7 [%] 










Had No Earntnga, Racaivad No AFOC 
Pay aanta 


7.9 


11.9 


-3.9 


0.166 


Had Boat Earnlnge, Racaivad No AFOC 
Pay aanta 


22.0 


19.4 


2.6 


0.525 


Had No Etrnlnga, Racaivad Soae AFOC 
Pay aanta 


41 .6 


43.1 


-1 .5 


0.769 


Had Soae Earnlnge, Received Soae 
AFOC Payaenta 


26.4 


25.6 


2.6 


0.536 


Total 


100.0 


100.0 


0.0 




Eaployaant and Welfare Status, 
Quarter 8 [X] 










Hed No Earnings, Received No AFOC 
Pay aanta 


10.6 


14.0 


-3.4 


0.304 


Had Soaa Eernlngo, Received No AFOC 
Pey eent e 


25.0 


22.9 


2.1 


0.627 


Hed No Earnlngei Received Soae AFOC 
Pay eent a 


39.2 


42.6 


-3.5 


0.493 


Hed Soae Earnings, Received Soaa 
AFOC Payeents 


25.2 


20.5 


4.6 


0.273 


Totel 


100.0 


1 00.0 


0.0 




Eaploynent and Welfare Statue, 
Querter 9 [X] 










Hed No Earnlnge, Received No AFOC 
Pey een te 


11 .4 


14.4 


-3.1 


0.366 


Hed Soae Eernlnga, Received No AFOC 
Peyaent e 


25.9 


25.3 


0.5 


0.905 


Hed No Eernlnga, Received Soee AFOC 
Pey eent e 


36.1 


42.2 


-4.1 


0.414 


Hed So»b Eerning8f Received Soae 
AFOC Peyeenta 


24.7 


16.1 


6.6 


0.122 


Total 


100.0 


100.0 


0.0 





( con ti n ued ) 



ERIC -185- 227 



TA8LE C.2 (continued) 



Outcone and Follow-Up Period 


Exparinentals 


Con tro I s 


D i ffe renca 


P 


Enploynant and Welfere Statue, 
Quarter 10 [X] 










Had No Earni.iga, Received No AFDC 
Pay sent a 


13.0 


21 .3 


-8.2»« 


0.028 


Had Some Earninge, Received No AFDC 
Paynan ta 


28.1 


26.3 


1.8 


0.696 


Had No Eernlngs, Received Some AFDC 
Paynan te 


36.2 


39.1 


-2.9 


0.565 


Had Some Earnings, Received Some 
AFDC Peyraents 


22.7 


13.4 


9.3«« 


0.023 


Total 


100.0 


100.0 


0.0 




Eaploynent and Welfare Stetue, 
Qua r ta r 11 [ % j 










Had No Earnings, Received No AFDC 
Pay men 1 8 


15.6 


18.9 


-3.3 


0.393 


Had Some Earnings, Received No AFDC 
Paynan ta 


28.6 


31.3 


-2.7 


0.557 


Had No Eernings, Received Some AFDC 
Payments 


34.1 


31 .9 


2.2 


0.648 


Had Some Earnings, Received Some 
AFDC Peyments 


21 .8 


17.9 


3.8 


D.3S7 


Totel 


100.0 


100.0 


0.0 




Seaple Size 


297 


147 







SOURCE AN0 NOTES: Sea Tebla 5.2. 
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APPENDIX D 
DECOMPOSITIO N OF IMPACTS 

This appendix explains the calculations reported in Table 5,8. Any 
full -sample cumulative earnings impact is the sum of three components. 
Part of the earnings impact is due to (1) the experimental-control 
difference in average rates ever employed. The other two components are 
due to experimental-control differences among those who ever were 
employed. One component arises from (2) the difference in the average 
number of quarters with employment; the other stems from (3) the 
difference in average earnings per quarter with employment. 

For the Maine TOPS evaluation, 297 experimentals and 147 controls 
were chosen at random. More generally, let n T treatment group members (T) 
and n c control group members (C) be chosen at random. The total sample 
size is n - n T + n c . Let 

A/, - 1 

if sample member i is employed, i - I n , and let it equal zero 

otherwise. Then 

AT r - [\/n T ] £ A/, 
■<<o 

is the fraction employed, and 1 - W T -bar is the fraction not employed, 
among those assigned to treatment. The absolute number employed among 
experimentals is 

«<r> 
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while the number not employed among experimentals is 



Similarly among control sample members, the proportion employed is 
M c - [\/n c ] £ M r 



y«c> 



and I - W c -bar is the fraction not employed. The absolute number 
employed among controls is 



while the number not employed among controls is 
n co ■ u c -n C] . 

Let y t » be the raw unadjusted earnings outcome for sample member i. 
The adjusted impact d is the coefficient of a duamy variable for 
irsmbership in the treatment group in the regression equation 

y,' - a + ds, + c'z t + e,. 

Using coefficients estimated from this regression, where z L is the 
ith sample member's vector of covariates measured before random 
assignment, and z-bar is the grand mean for these covariates, 
covariate-adjusted outcome scores are defined as 

y. - y/ - z). 

Then 
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y r ■ (i/rc r ) Y. y> 

is adjusted moan earnings in Che treatment group, and 
y c « {\/n c ) Y. y, 

is adjusted mean earnings in the control group. The quantity 

<* * yr - y c 

is the adjusted sample impact of the treatment. Its value is precisely 
the same as the value of the coefficient of the dummy variable in the 
regression equation; see Ostlc (1975, p-. 215, 461). 

Earnings are zero for any sample member who is not employed. This 
observation allows adjusted mean earnings for controls to be expressed as 




lei 1 v 



Similarly, mean adjusted earnings among experimental may be 
expressed as 

y r = U T y TtUml . 
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The adjusted impact is 

d = y T - y c 

This expression is very similar to expression (7) in Oaxaca (1973). 1 
Two alternative decompositions of such an expression carried out by Oaxaca 
may be combined very easily. Where V is any fraction between zero and 
unity, 

d -v(^ T y T3/J . l --M c y c , Uml ) + (\-v)(-M T y T3Uml -B c y C3i/ . l } 

- y{^ T y T3 u.i + ^ T y C3l j.i - ^ry C3 v-i - ^ c yc 3W .i! 
= v{ttf T {yT,u.i-yc 3 u.i) + yc,>j. l {'M T -M c )} 

+ ( 1 - V)(M c {y T3U . l - y CiA ,.,) + y T „j-i{M T - M c )} 

+ \- V yc 3M .i{M T -M c ) + (l-V)y T3Mml (M T -M c )] 

= [{K^ r + (l-K)¥ c >(y r3W . 1 -y C3W . 1 ) ] 
+ [<Vyc,u- l +{l-V)y T3Mml }(M T -M c ) ] . 

The last member of this expression is composed of two bracketed 
terms. The first bracketed term is the amount of the adjusted earnings 
impact explained by differences in earnings while employed. It is the 
experimental -control difference in adjusted earnings while employed, . 



Donald Oaxaca. "Male-Female Wage Differentials in Urban Labor Markets." 
International Economic Review 14, 3 (October 1973): 693-709. 




weighted by the average employment rate. The second bracketed term is the 
amount of the adjusted earnings impact explained by differences in rates 
of employment. It is the difference in employment rates, weighted by 
average adjusted earnings while employed. 

Thus average earnings, which is the product of two factors, has an 
impact which may be expressed as the sum of two terms , one for each 
factor. The first factor is the average amount of earnings while 
employed. The second factor is the average rate of employment. 

Similarly, the amount of earnings while employed may be expressed as 
the product of two factors, and the experimental-control difference in 
earnings while employed may be expressed as the sum of two terms. For 
those who ever were employed, the average number of quarters with 
employment, k-bar, and average earnings per quarter with employment, 
v-bar, are the two factors. The second factor is defined by 

and 

The two terms of the decomposition become 

y 73^-1 -ycBtf-i = % T W T - Jc c W c 

- [{Vlc T + (l-V)]c c }{w T -w c )] 
H{Vw c + {1-V)w t }{Jc t -Jc c )]. 

When the two decompositions are put together into one expression, the 
full-sample impact on earnings becomes the sum of three terms: 
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d = {K^ T + (.. -V)M c }[{V1c t + (1-V)1cc}{wt-*c) 
+ <Vw c +(l-V)Ui T }(K T -K c )] 

= [<VB T + (l-V)M c }{Vlc T + (l-V)Tc c }(w T -w c )] 
+ [{VM T + (l-V)-M c }{Vw c + (\-V)w T }[K T -K c }] 
+ [{VK c Ui c +(l-V)K T U) T }(-M T --M c )]. 

The first term is the amount of the full-sample earnings impact due 
to the experimental -control difference in average earnings per quarter 
with employment, among those who ever were employed. The second term is 
the amount due to the difference in the average number of quarters with 
employment, among those who ever were employed. The third term is the 
amount due to the difference in average rates ever employed. When divided 
by d, the full-sample impact, these three terms become, respectively, the 
proportions in columns two, three, and four of Table 5.8. Parameter V was 
given the value 0.5, so thet Oattaca's two basic alternative decomposition:; 
were weighted equally. 

A good example to illustrate the decomposition -..f impacts with this 
formula is the first line of Table 5.8. Parameter d - y„ r bzr - y c -bar in 
this case is the cumulative impact on earnings from Quarters two through 
eleven. Table 5.3 shows that d - $1,744.83, y T -6ar - $7,344.00, and 
y z -bar - $5,599.17. From the sample sizes in Table 5.7, M 7 -bar - 243/297, 
and W c -5ar - 118/147. Cross -tabulations of numbers of quarters with 
employment show that, among those ever employed, k 7 -bar - 4.6296 / H 7 -bar, 
and V&ar " 4.?197 / M c -bar. This means that 

y r ,«.i = y r /^r " $8,976.00; 
y C ,«.i - y c /^c - $6,975.24; 
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= y r >w-i /E r 55 SI. 586, 31; 

u>c = yc>A/-|/fc 55 $1,296,19, 

Substituting these values into the last formula above shows that the 
amount of the cumulative earnings impact due to the difference in 
cumulative employment rates jas $123.31; the amount due to the difference 
in number of quarters with employment, among those ever employed, was 
$323.67; and the amount due to the difference in average earnings per 
quarter with employment, among thooe ever employed, was $1,297.88. 
Dividing each of these amounts by $1,744.83 produces the entries in the 
first row of Table 5.8. 
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APPENDIX E 
ESTIMATING IMPACTS OF RECEIVING SERVICES 



I. I ntroduction and Summary 

This Appendix discusses the methodology and findings for analyzing the 
TOPS impacts for those participating experimentals actually receiving 
services. Given a number of key assumptions, participant impacts can be 
derived fran overall experimental impacts presented in the text by multi- 
plying them by an adjustment r actor. Because rates of participation by 
experimentals were quite high in TOPS, impacts of actually receiving ser- 
vices are fairly close to the impacts of assignment to services presented 
in 'che text? a reasonable value for the adjustment factor is 1.12. Thus 
the adjustment is not crucial for evaluating TOPS. However, an explanation 
of the adjustment may prove useful in planning and evaluating similar 
programs. 

II. The Impact of Participation in TOPS 

Chapter in explained in great detail the services available io the 
297 experimentals in the research sample, it was also made clear that not 
everyone who was assigned to experimental status actually participated in 
all or oven in any of the components of TOPS. In particular, only 91 of 
the 297 experimentals actually received the subsidized job which was the 
key element of TOPS, and 10.4 per cent of assignees never received any TOPS 
services. However, everyone assigned to experimental status was included 
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when calculating average impacts of assignment to TOPS* 

Just as impacts of assignment do not take into account nonparticipa- 
tion by experiment als, they also do not adjust for participation in TOPS- 
like activities by controls. As explained in Chapter IV, none of the 147 
controls received the complete sequance of pre-vocational training, field 
placement, and wage subsidy which constituted the entire TOPS treatment, 
Hcwever, controls were eligible for an array of WEET services which 
included TOPS-like ccmpnents. Table 4.2 shows that 33 controls received 
services similar to one or two of the components of TOPS. 

In Table 4.2, it was pointed out that not all experimentals received 
the TOPS treatment; 266 (or 89,6 percent) received one, two or three of the 
components while 31 (or 10,4 percent) received none. The average impact 
for experimentals is thus a weighted average of two parts. One component, 
which gets a weight of 0,896, is the average impact among those who 
received TOPS. The other component, which receives a weight of 0,104, is 
the mean impact among those who did not _eceive TOPS. 

Similarly, it was pointed out in Chapter IV that 33 (or 22.4 percent) 
of the 147 controls participated in one or two TOPS-like activities. Thus 
114 ;or 77.5 percent) participated in no TOPS-like activities. The average 
impact for controls 's a weighted average of two components. One compon- 
ent, which receives a weight of 0.224, is the mean impact outcome among 
those controls who received sane TOPS-like service. The other component, 
which is weighted by 0.776, is the mean impact among those who did not 
receive TOPS-like services. 

An impact of assignment is the difference between average outcomes for 
those assigned to the experimental group and those assigned to the control 



ERLC 



group. Impacts of assignment would Lo much easier to interpret if all of 
the experimental had received TOPS, and if none of the controls had 
received TOPS-like services. In such a simple state of affairs, all of the 
average impact of TOPS would have been spread evenly over the experimental 
group, and TOPS-like activities would have had no effect on the control 
group. However, it has already been seen that experimental and control 
behavior was more complicated. The basic problem here is that four out- 
comes are available for estimating one impact of receiving services. 
Average outcomes are available for those who did and did not receive TOPS 
or similar services, in both the experimental and the control groups. The 
impact of assignment is the difference between the impact on experiment als 
and the impact on controls, and each impact has two components. 

One simple solution to this problem is to make assumptions about the 
components. Among experiment als, suppose that the impact of TOPS on those 
who did not receive it is precisely zero. This would mean that any impact 
of participation is concentrated entirely among the 89.6 percent of 
experimental who did receive TOPS. Furthermore, suppose that, among the 
22.4 percent of controls who received some TOPS-like services, there was an 
impact of participation exactly M times the impact among the participants 
in the experimental group. 

These assumptions allow tne impact of assignment to TOPS to be 
reinterpreted: 



Average impact of assignment 



= Average impact among all expert mentals 



- average impact among all controls 



= 0.C96 (average impact of participation) 




+ 0.104 (zero impact on nonparticipating experimentals) 

- M (0.224) (average impact of participation) 

- 0.776 (zero impact on nonparticipating controls) 
■ (0.896 - 0.224M) (average impact of participation). 

Parameter M is the effectiveness of participating for controls rela- 
tive to the effectiveness of participating for experimental s. 1 According 
to the last term of this equation, when parameter M is unity, each average 
impact of assignment to TOPS reported in the tables presented above is 
0 .672 times the average impact of participation i;i TOPS. To get the 
corresponding impact of participation, it is ner ^ssary only to divide the 
impact of assignment by 0.672; that is, to multiply it by (1/0.672) - 1.49. 

III. Assumptions 

Two assumptions were made in order to calculate the adjustment factor 
of 1.49 for getting impacts of receiving services. First, it waa assumed 
that impact! on nonpar ticipants* were precisely zero. Second, it was 
assumed that the impact on participating controls was the same as the 
impact on p 'ticipating experimental (M=l). Hcv reasonable are these 
assumptions? 

Among controls and among experimentals, participation was defined as 
receiving any TOPS or TOPS-like services. Only if a sample member received 
none of the three ccmponents was she considered a nonpar ticipant. If sanc- 
tions had been applied tc such nonparticipants, one might expect sraa^er 
AFDC income and perhaps greater rates of employment to replace AFDC it 
lost to sanctions. Even the mere threat of sanctions might have strong 
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deterreat effects on earnings and welfare income. Since sanctions only very 
rarely followed failure to appear for prevocational training, and since 
there was no evidence of long-term TOPS contact with nonparticipants, it 
s'eents reasonable to assume TOPS had no effect on experimentals who received 
none of its components. Since there was no post- assignment TOPS contact at 
all with nonparticipating controls, it seems reasonable to assume TOPS had 
no effect on nonparticipating controls. 

The second assumption is much more problematic. For example, none of 
the controls, but 27.3 percent of experimentals, received all three 
components of TOPS. Moreover, Chapter VI pointed out that the TOPS-like 
services received by controls were cheaper and less intensive than TOPS 
services. Apparently, the average treatment received by those 33 controls 
who got Tops-like services was weaker than the average TOPS treatment 
received by those experimentals who got TOPS. Thus it is unlikely that the 
effect of TOPS-like services on participating controls was as large as the 
effect of TOPS on participating experimentals. 

If the second assumption is dropped, and it is assumed instead that 
the effect of TOPS-like services on participating controls is zero, then 
the arithmetic presented earlier changes, so that <Uie adjustment factor is 
(1/0.896) « 1.12. Since this alternative assumption may be a little weak, 
an overall sensitivity analysis of the adjustment might conclude that, to 
calculate an impact of participation, the impact of assignment should be 
multiplied by a faotor somewhere* between 1.12 and 1.49. 2 

An alternative approach is to test the assumption o£ a zero effect on 
nonparticipants directly. As discussed in Cave, 1987, participation may be 
modeled, and predicted participation used to separate controls as well as 



experimentals into groups for comparison of outcomes. However, the success 
of such an approach depends on the success of the participation model and 
may be sensitive to the particular functional form chosen for model error. 

IV- Random Variation in Component Weights 

Besides its assumptions about the magnitudes of impacts in sane groups 
of experimental and controls, the adjustment procedure just presented has 
another basic weakness. Randomness in the weights given subgroups of 
experimentals and controls was not takeft into account. Every member of the 
target population for the TOPS program is a potential member of one of the 
four groups described and given the weights indicated above, if a differ- 
ent sample of 444 experimentals and controls had been drawn, the weights 
used might have been quite different, because different proportions of the 
four groups might have been selected. 

In the equation presented above, the impact of receiving services is 
the rc\io of the impact of assignment to an expression involving two of the 
weights. since the numerator and the denominator of this ratio are 
correlated, the effect of participation is a ratio of correlated random 
variables. Proper point estimators and standard errors of such a ratio 
cannot be obtained simply by dividing one sample value by another. 3 a 
modified ratio estimator presented in Cave (1988) remedies this problem. 
The approximate variance of this modified ratio estimator is the same as 
the approximate variance of the crude ratio estimator presented above. For 
the TOPS sample of 444, the two estimators are indistinguishable. When 
M=0, the modified estimate is 1.12, with a standard error of 0.C22. When 
M*l, the point estimate is 1.49, and the standard error is 0.086. 
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V. Summary 

When all those assigned to a program treatment actually receive it, 
and none of those assigned to the control group receive similar treatment, 
there are only two groups of sample members to consider. The program 
impact in such a situation is simply the difference between average out- 
cccaes for the two groups. In practice, there are four groups of sample 
members to consider. In the Maine TOPS evaluation, while 89.6 percent o£ 
experimental received at least one TOPS component, 10.4 percent of experi- 
mental did not receive any of its components. While 77.6 percent of 
controls received no TOPS-like service, because such services were part of 
an array of services intended to be offered to controls, 22.4 percent of 
those assigned to the control group did receive at least one TOPS-like 
service. 

The impacts of assignment to TOPS presented in the text are unbiased 
differences in average outcomes between all assigned to the program and all 
assigned to the control group. The presence of two groups of experimental*: 
and two groups of controls makes the observed difference in average outcome 
between all experimental and all controls interpretable as the difference 
between two impacts: the impact on experimental, and the impact on 
controls. To transform the observed impact of assignment into an impact of 
participation, the observed impact may be multiplied by a rough adjustment 
factor. For the TOPS evaluation, a reasonable value for this adjustment 
factor is 1.12. 
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CHAPTER I 



1. Auspos et al M 1.985 • 

2. The Work Opportunities Committee published its report and 
recommendations in Women, Work and Welfare , September, 1981. 

3. Work Opportunies Ccwmittee, 1981, p. 15. 

4 . This component, was operated under the rules of WIN Work 
Experience rather than those of the Community Work Experience 
Program (CWEP). CWEP specifies that the number of hours 
worked at a worksite per month by a participant must be equal 
to the AFDC recipients 1 grant divided by the minimum wage. 

5. Data on the September 30, 1983 caseload were provided by the 
Division of Welfare Employment, Maine Department of Human 
Services. Data on WEET service prevision in federal fiscal 
year 1984 (October 1, 1983 - September 30, 1984) are published 
in *the Division of Welfare Employment* s Service Delivery Plan 
for FFY 1985. The latter data count persons active in more 
than one activity and the total number active for each 
activity. 

6. For a summary of the findings in other states see Gueron, 
1987 . For individual state reports, see the list of MDRC 
publications at the end of this report. 

7. MDRC Board of Directors, 1980 . 

8. Arizona, Florida, New Jersey, Texas and Vermont also received 
OFA funds. MDRC has published two reports on the implementa- 
tion of grant diversion programs in these states and Maine, 
and is also conducting an impact and benefit-cost evaluation 
of the New Jersey program. See Bangser et al. , 1985 and 1986. 



CHAPTER II 

1. For variations, see Table 1.1. JTPA staff 'ere also involved 
in assessment interviews in some cycles in jjewiston, Augusta, 
Bath and Bangor. 

2. Bane and Ellwood, 1983 « 

3. This sample size dif/.ers, by 12, from the total number of 
enrollees stated in the report on interim findings as randomly 
assigned between October 1983 and December 1984. That sample 
size was based upon the nunber of randan assignments called in 
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to MDRC, Eleven cases included In that first report total 
were either repeat random assignments ~ one person assigned 
more than once ~ or cases with irreparable data problems* 
These random assignments were subsequently removed from the 
sample. Also, one sample member was male and was therefore 
taken out of the sample* 

4, In fiscal year 1982, 45 percent of new WEET registrants were 
exempt from the mandatory registration requirement; in fiscal 
year 1983 , the proportion rose to 53 percent, according to 
data provided by the Division of Welfare Employment, 

5, The 16,556 AFDC cases in this January 1983 profile consist of 
recipients in Maine's AFDC program, excluding refugees in the 
Resettlement Irogram, recipients with grants less thau $10 per 
month, and recipients not receiving an AFDC payment during the 
month of January 1983 , 

6, In January 1983 and again in January 1985, the Maine Depart- 
ment of Human Services compiled descriptive data on the AFDC 
caseload in these study months, The first study month showed 
less than a third of the caseload to have received AFDC for 
more than the 24 consecutive months prior to this study month. 
In January 1985, almost one half of the caseload had been 
receiving AFDC for more than the 24 consecutive months prior 
to the study month. it would appear that the Maine caseload 
was shifting towards having more long term recipient!;. The 
random assignment period for this demonstration fell between 
these two study months, and the proportion of the sample that 
received AFDC for more than the 24 consecutive months prior to 
random assignment is between these two figures? 32 percent. 
This increase in the number of long term recipients coincides 
with the changing eligibility rules in Maine, as described in 
Chapter IV. 

7, Calculated from the MDRC Client Information Sheets and uata 
provided by the Division of Welfare Employment on the WEET 
caseload as of April 1, 1984. The TOPS sample includes 425 
individuals; data for the remainder of t sample were 
missing. The WEET registrant caseload as of April 1, 1984 
included 7,432 individuals; educational data were available 
for only 7,186 of these individuals, 

8, See Footnote 4, 

9, Sixteen sample members did not complete the CIS at all. For 
these enrollees all the demographic variables from this source 
will be missing. For five other sample menbers, a few speci- 
fic pieces of demographic information are missing. For the 
impact analyses, the modal values for similar sample monbers 
(that is, eniwilees with the same employment and welfare 
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receipt history within the same region) were substituted for 
these missing observations. 



10. The reliance on administrative records to measure outcomes 
offers many advantages as well as sane limitations. Since 
adninistrative records do not require ongoing contact with 
sample members, they are a less expensive way to collect data, 
and result in fewer missing observations in the later follcw- 
up periods. Adninistrative records also do not depend on the 
ability of individuals to recall precise but im^urtant infor- 
mation, such as dates, earnings or the length of enrollment in 
program activities. However, administrative records are 
limited in the types of outcomes that they measure. Issues of 
quality and completeness will be addressed within the 
discussion of each source* 

11. Calendar quarters are three mon< h groups beginning with 
January of each year. Quarter 1 is January, February and 
March? quarter 2 is April, May and June; quarter 3 is July, 
August and September? and quarter 4 is October, November and 
December. 

12. First, employment data reported by the state UI system was 
compared with self-reported employment prior to random assign- 
ment fron the CIS. Thirty-three percent of the 12 sample 
members who reported being employed more than 18 months in the 
two years prior to random assignment did not have Ul-reported 
earnings in the year prior to random assignment. This figure 
was slightly higher than the percentages calculated frcm the 
same comparison in other states in MDRC's Demonstration of 
State Work/Welfare Initiatives. Second, job placement data 
frcm the WEET tracki^" system was compared with UI- reported 
earnings. Two percent of the 47 sample members who were 
randomly assigned in the last month of a calendar quarter and 
placed witchln 12 months after randan assignment, did not have 
earnings reported to the UI system in the second through fifth 
quarters after random assignment. 

13. The aggregation of monthly AFDC payments into calendar quarter 
amounts is unlike the measures used in other states in MDRC's 
Demonstration of State Work/Welfare Initiatives. In other 
studies, the AFDC monthly grants were summed into three month 
totals starting with the month of random assignment. 

14. Automated payment systems are not generally intended to record 
all payments actually made to welfare recipients. There were 
occasions when checks were hand-written to clients. It was 
therefore necessary to determine whether the research data 
were considered sufficiently complete to estimate program 
Impacts. 
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The automated data for 41 cases were compared with the case- 
files for these cases. In total, 188 months of data were 
checked. In 88 percent of these months of data, the payment 
on the automated file matched the casefile records. The 
remaining unconfirmed months were all discrepancies of dollar 
amounts (i.e., there was a recorded payment in the casefile, 
but the check written from the automated file was for a 
different amount) except one month where there was a check 
written and, according to the casefile, one should not have 
been written. 



CHAPTER III 

1. Auspos et al., 1985. 

2. Auspos et al., 1985, p. 49. 

3. These reasons are based on participation logs kept by WEET 
staff, see Auspos et al., 1985, pp. 47-48. 

4. The average total time to complete the TOPS sequence, for 
experimental who entered all three components, was estimated 
from WEET Information System data and supplemental OJT 
completion information supplied by WEET staff. The average 
number of months between random assignment and the start of 
CXJT (7.1 months) was added to the average actual duration of 
the CJT (16.8 weeks) for a total of 11 months until the end of 
participation. 

5. Auspos et al. , 1985, p. 72. 

6. The three locations were Bath, Damariscotta and Portland. 
Auspos et al., 1985, p. 64. 

7. The interviews are analyzed in detail in Auspos et al., 1985, 
pp. 65-69. 

8. For the 118 experimentals who entered work experience but did 
not continue into OJT, employment and AFDC statuses after 
participation in work experience were assessed. Three months 
after the start of work experience was considered the month 
work experience would have been completed, if completed 
according to schedule. Earnings are measured on a quarterly 
basis. Therefore, if this month was the first month within a 
calendar quarter (i.e., January, April, July or October) 
employment was indicated by earnings within that quarter. 
Otherwise, employment was indicated by earnings in the 
following quarter. 
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9. Between October 1983 and October 1984, Maine's JTPA system 
placed close to 200 AFDC recipients in OJT (including TOPS 
participants), and WEET placed an additional 50 registrants in 
0JT. Auspos, et. al. , 1985, p. 51. 

10. The minimum wage in Maine rose from $3.35 an hour to $3.45 an 
hour in January 1985, and to $3.55 an hour in January 1986. 

11. Auspos et al., 1985, p. 80. 

12. Auspos et al. , 1985, p. 76. 

13. Bangser et al., 1986. 

14. Reasons for non-canpletion and length of stay information were 
calculated by MDRC from WEET- reported data. 

15. A detailed analysis of the interviews is in Auspos et al. , 
1985, pp. 79-83. 

16. The process of estimating subsidy levels in grant diversion 
programs is discussed in Bangser et al., 1986. The actual 
amount of AFDC payments diverted into the wage pool for each 
sample member participating in OJT were obtained fran WEET 
Grant Diversion Reports for the period frcm April 1984 through 
July 1986. These amounts were summed for all cases across 
this entire time period. 

An estimation of the total wage subsidy owed to employers was 
calculated from wage and hours information in the TOPS OJT 
contracts, and supplemental dat? on OJT completions supplied 
by WEET staff. This subsidy was calculated as: 0.5 (OJT 
starting hourly wage x OJT actual length- of- stay x OJT 
scheduled hours per week) « Actual Payments to Employers. 



CHAPTER IV 

1. Bane and Ellwood, 1983. 

2. This information is based on: Bureau of ths Census, U.S. 
Department of Commerce, 1987, p. 393, Table 668; Bureau of the 
Census, U.S. Department of Commerce, 198.1, Part 21, Maine, 
Vol. 1, Chapter C, p. 18; and Ross and Danziger, 1984, Table 
1. 

3. Bureau of the Census, U.S. Department of Commerce, 1987, p. 
393 , Tabl* 668; and Ross and Danziger, 1984 , Table 1. 

4. Work Opportunities Committee, 1981, p. 15; and Levitan and 
Shapiro, 1987. 
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5 * AFDC Sta tistical Resource; Selected Characteristics of Cases 
and Persons Receiving AFDC Benefits , 1983, Table 16, 



6. The Federal Deficit Reduction Act (DEFRA) of 1984 modified the 
GBRA previsions in several key areas, DEFRA raised the 9ross 
income test from 150 percent of need to 185 percent of need. 
It also replaced the four-month limit on the $30 disregard 
with a twelve-month limit, and extended the $75 work expense 
disregard to part-time workers as well as full-time workers, 

7. Countable income includes both unearned income and countable 
earnings. 

8 • Character istics of State Plans for Aid to Families with 
Dependent Children , 1985, pp. 337-8, and 1987, pp. 402-3. 

9. Maine Public Assistance Payments Manuals* 

10. Office of Family Assistance, U.S. Department of Health and 
Human Services, 1987, pp. 402-3. 

11. Communication with Bureau of Labor Statistics, 1987. 

12. Communication with Maine Department of Labor, 1987 . 

13. Separate estimates of the cost of pre-vocational training for 
controls were not available. 



1. Average experimental and control group outcomes reported here 
are adjusted means from an analysis of covariance procedure 
controlling the difference in outcomes for 14 individual 
characteristics before random assignment. See Ostle, 1975, p. 
461 and Appendix Table C.l. 

2. When an impact is statistically significant, if the effect of 
the program on its target population really were zero, a 
difference as large as that observed in the sample would occur 
by chance no more than 10 percent of the time. Thus a statis- 
tically significant expertoental-control difference can be 
generalized beyond the particular sample drawn for the evalua- 
tion to draw inferences about the effect of the program on its 
target population. 

3. The minimum sample size needed for statistical significance 
increases as the true program effect becomes smaller, as the 
variance of the outcome increases, as the R-square of the 
impact equation declines, as the sample split between experi- 
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mentals and controls becomes more uneven, as covariate differ- 
ences between groups increase, and as the statistical signific- 
ance level decreases. See Cave, 1987 ♦ 

4, There is a statistical procedure that can yield an adjustment 
factor that will convert impacts of assignment to services- to 
impacts of receiving services* For TOPS impacts this adjust- 
ment factor is 1.12. See Appendix E for the derivation. 

5. True impacts on those placed and not placed in subsidized jobs 
would entail comparisons between members of each experimental 
group and their unknown counterparts in the control group. It 
is possible that true impacts on those placed in OJT could be 
positive for welfare outcomes and negative for labor market 
outcomes. Hcwever, if true impacts on those placed in OJT had 
the same signs and approximate magnitudes as the experimental- 
subgroup differences just cited, a plausible explanation of 
the overall AFDC finding would emerge. Effects opposite in 
sign and about twice as strong for those placed in OJT slots 
tend to counterbalance effects on those not in an OJT 
position, because the third of the sample placed in OJT slots 
is weighted half as heavily in overall impacts. 



CHAPTER VI 

1. Many of the techniques used in the analysis were first develop- 
ed for the Supported Work evaluation, the Employment Opportune 
ity Pilot Project evaluation, and evaluations of the other 
state programs in MDRC s Demonstration of State Work/Welfare 
Initiatives, See Kemper, Long and Thornton (1981) and Long 
and Knox (1985) for additional information. Details on the 
data and methodology underlying all estimates and all 
projections in this chapter are available frcm the authors. 

2. OJT wage costs were calculated as the OJT contract wage 
multiplied by OJT contract scheduled hours per week, the 
product of which was multiplied by the actual length-of-stay 
according to WEET program staff records. For the 10 OJTs 
without contracts, the average total OJT wage (according to 
case records for a sample of 66 controls) was used. 

3. The data, which are based on a national survey of firms, are 
reported in Employee Benefits 1985 (Washington: U.S. Chamber 
of Commerce. 1986). It should be noted chat the exclusion of 
payroll taxes frcm this rate ( tc is treated under 'Tax 
Payments." below) is a change frcm past MDRC evaluations of 
welfare employment programs. 

4. In most OJT assignments employers paid full compensation — 
that is, 100 percent of wages and fringe benefits — to the 
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TOPS participants when they completed their assignments and 
were hired as regular employees, which suggests that tL- value 
of their work during the assignments was close to (if not more 
than) the full conpensation level. In the work experience 
assignments, agencies indicated that the work done by TOPS 
participants was usually important to the agency, indicating 
that the value of their work was much greater than zezo, 
although probably less than the full compensation level. For 
additional discussion, see Kemper and Long, lf/81. 

5. For work experience positions, the wage and fringe benefit 
rates used were what the employer would have paid a regular 
worker in that position. For CUT positions, the wage used was 
the average starting wage peid by the employer according to 
the OJT contracts. The fringe benefit rate used was the 
average paid by the employer according to the worksite survey. 

6. The estimation of federal taxes used 1S85 tax rates and 
exemptions for the observation period, since these applied to 
wost of tha earnings obtained during this period, and 1987 tax 
^ates for the projection period, since most of the 
post-observation period falls under the newer tax laws. For 
state and local taxes, 1985 tax rates and regulations were 
used. 

7. Food Stamps regulations dictate eligibility and benefit levels 
based on household income, where deductions are allowed for 
out-of-pocket work-related expenses such as child care, 
medical expenses and shelter costs. Within constraints of the 
available data, estimates of the value of Food Stamps have 
been mad using U.S. Department of Agriculture procedures for 
calculating total income and total allowable deductions. Seme 
of the data used in making these calculations were provided by 
the Maine Bureau of Food Stamps, Department of Human Services. 
It has been assumed that 80 percent of AFDC recipients 
actually receive Food Stamps, based on an unpublished 
tabulation made by the Department of Agriculture usi i the 
1984 Survey of Income and Program Parti .ipationn (SIPP) panel. 

8. OJT participants were considered on welfare, and under the 
normal Medicaid eligibility rules, even if they were not 
receiving a regular case grant due to OJT earnings. Until 
October 1984 , an individual was eligible for Medicaid for four 
months after going off welfare. Subsequent changes in the 
regulations required states to provide nine months of Medicaid 
coverage to former AFDC recipients who lost their AFDC 
eligibility due to termination of the earnings disregard 
(i.e., the amount of earnings that had previously been 
excluded from the benefit calculation) . For the benefit-cost 
analysis the estimated program effects on Medicaid credited 
persons as being eligible for Medicaid if they qualified 
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either under the four-month rule or the nine-month rule. 

9. Data used to calculate the average value of Medicaid payments 
for AFDC recipients were obtained frcn the Maine Department of 
Human Services, Office of Data and Research, for January 1983. 
This estimate has been adjusted for inflation using the GNP 
price deflator to reflect 1985 dollars, the base year for all 
of the final results of this analysis. 

10. The state administrative costs of AFDC and Medicaid were 
estimated as the product of the estimated per capita costs of 
administering these programs for one month during July 1984 
through June 1985 and experimental-control differences in the 
number of months of eligibility. Federal administrative costs 
for each of these transfer programs were estimated on the 
basis of the ratio of total federal administrative 
expenditures to total federal transfer payments. Data for 
estimating the monthly administrative costs of AFDC and 
Medicaid were obtained from the Maine Department of numan 
Services, Divisions of Medicaid Policy and of Financial 
Services and the Appendix to the Federal Budget for Fiscal 
Years of 1986 and 1988. Unemployment compensation and Food 
Stamps administrative costs were estimated by multiplying the 
experimental-control differences in t^e average value of 
compensation payments received by the ratio of combined state 
and federal administrative costs to total benefits during FY 
1985; data were obtained from Maine Bureau of Food Stamps, 
Department of Human Services, and Appendix to the Federal 
Budget. 

11. The cost of n?itle XX child care used by TOPS clients was 
calculated with information from WEET about the proportion of 
it's clients receiving care funded by Title XX, and infonna- 
tion from the Bureau of Social Services Planning and 
Evaluation Unit on the average length of care and the average 
cost per week. 

12. in these 12 tests, one of the assumptions used to calculate 
the benchmark estimates was altered (keeping all other 
assumptions unchanged). The assumption regarding the discount 
rate, which as an annual real rate of 5 percent for the 
benchmark estimates of net present value, was changed to zero 
percent in one test and to 10 percent in a second. The decay 
rate assumption — zero for the benchmark estimates — was 
changed to an annual rate of 22 percent for one test and 
infinity (equivalent to assuming there are' no future effects) 
in another. The base period estimates, which were the impacts 
measured in the last two quarters of the observation period 
for the benchmark estimates, was changed in several alterna- 
tive ways: (1) the base was increased by 50 percent above what 
was measured for all types of impacts; (2) the base was 
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reduced by 50 percent for all impacts? (3) the base was 
increased by 50 percent for employment-related impacts only 
(that is, earnings, taxes, UI and Pood Stamps); (4) the base 
was increased by 50 percent for welfare-related impacts only 
(AFDC and Medicaid); (5) the base was reduced by 50 percent 
for welfare impacts only; and (7 ) nvaw base estimates were 
constructed using the last five available quarters of 
observation, where the last quarter was given the most weight 
and each prior quarter was given sucessively less weight. 
Finally, this last base estimate was tested in combination 
with an annual decay rate of 22 percent. 



CHAPTER VII 

See MDRC's findings on the AFDC target group in the National 
Supported Work Demonstration, MDRC Board of Directors, 1980. 

Bangser et al., 1985 and 1986. 



APPENDIX E 



Parameter M is the effectiveness of participating for controls 
relative to the effec xveness of participating for experiment- 
als. On che one ham., this ratio reflects the extent to which 
controls get services similar to those obtained by experiment- 
als, whether they receive the equivalient of full treatment or 
only part of it. On the other hand, it also reflects self- 
selection among controls, and selection of controls by program 
operators; controls who participate may be those for whan the 
treatment is especially effective (M>1), or especially inef- 
fective (M<1). When M«l, the treatment is just as effective 
for the average participating control as for the average 
participating experimental, Absent special selection effects, 
in most circumstances results for M=0 and for M=l should 
bracket the true participation adjustment factor. 

A further complication is that controls who participated may 
not be a randan cross-section of controls but rather a select- 
ive group for whan the program impact is greater or smaller 
than average. If participating controls have larger impacts 
than average, the proper adjustment factor is larger than 
1.49; if participating controls have smaller impacts than 
average, the proper adjustment factor is smaller than 1.49; if 
participating controls have impacts opposite in sign to aver- 
age impacts, the proper adjustment factor is smaller than 
1.12. 

See Cave, 1988, In small samples, the ratio estimate is biased 
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by an amount proportional to the correlation between the 

numerator and the denominator. The standard error of the 

ratio involves this same correlation and the variance of the 
denominator as well as the variance of the numerator. 
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